GALE Arabic-English Word Alignment -- Broadcast Training Part 2

Introduction GALE Arabic-English Word Alignment -- Broadcast Training Part 2 was developed by the Linguistic Data Consortium (LDC) and contains 215,923 tokens of word aligned Arabic and English parallel text enriched with linguistic tags. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program. Some approaches to statistical machine translation include the incorporation of linguistic knowledge in word aligned text as a means to improve automatic word alignment and machine translation quality. This is accomplished with two annotation schemes: alignment and tagging. Alignment identifies minimum translation units and translation relations by using minimum-match and attachment annotation approaches. A set of word tags and alignment link tags are designed in the tagging scheme to describe these translation units and relations. Tagging adds contextual, syntactic and language-specific features to the alignment annotation. Other releases available in this series are: GALE Chinese-English Word Alignment and Tagging Training Part 1 -- Newswire and Web (LDC2012T16) GALE Chinese-English Word Alignment and Tagging Training Part 2 -- Newswire (LDC2012T20) GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web (LDC2012T24) GALE Chinese-English Word Alignment and Tagging Training Part 4 -- Web (LDC2013T05) GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 1 (LDC2013T23) GALE Arabic-English Word Alignment Training Part 1 -- Newswire and Web (LDC2014T05) GALE Arabic-English Word Alignment Training Part 2 -- Newswire (LDC2014T10) GALE Arabic-English Word Alignment Training Part 3 -- Web (LDC2014T14) GALE Arabic-English Word Alignment -- Broadcast Training Part 1 (LDC2014T19) Data This release consists of Arabic source broadcast news and broadcast conversation data collected by LDC from 2007-2009. The distribution by genre, words, tokens and segments appears below: Language Genre Files Words Tokens Segments Arabic BC 369 97,514 129,233 7,941 Arabic BN 40 70,635 86,400 3,752 Totals 409 168,149 215,923 11,693 Note that word count is based on the untokenized Arabic source, and token count is based on the tokenized Arabic source. The Arabic word alignment tasks consisted of the following components: Normalizing tokenized tokens as needed Identifying different types of links Identifying sentence segments not suitable for annotation Tagging unmatched words attached to other words or phrases Samples Please view the following samples: English Raw English Token Arabic Raw Arabic Token Word Alignment Sponsorship This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. Updates None at this time. Portions © 2008-2009 Al Arabiyah, © 2008 Abu Dhabi TV, Al Alam News Channel, Al Baghdadya TV, Al Hiwar, Aljazeera, Al Sharqiya, Dubai TV, Nile TV, Oman TV, PAC Ltd, Saudi TV, Syria TV, © 2008-2009, 2014 Trustees of the University of Pennsylvania

### 引言
GALE阿拉伯语-英语词对齐——广播训练第二部分由语言数据联盟（Linguistic Data Consortium, LDC）开发，包含215,923个经词对齐的阿拉伯语与英语平行语料Token，并富集了语言标注标签。该语料曾作为训练数据应用于美国国防高级研究计划局（Defense Advanced Research Projects Agency, DARPA）的GALE（Global Autonomous Language Exploitation，全球自主语言开发）计划。
在统计机器翻译的诸多研究路径中，一类方法会将语言知识融入词对齐语料，以此提升自动词对齐效果与机器翻译质量。该目标通过两种标注方案实现：对齐标注与语言标注。对齐标注依托最小匹配与依附标注手段，识别最小翻译单元与翻译关联关系；标注方案中设计了一套词标签与对齐链接标签，用于描述上述翻译单元与关联关系。标注环节可为对齐标注补充上下文、句法及语言专属特征。
本系列的其他发布版本包括：
1. GALE中英词对齐与标注训练第一部分——新闻专线与网络（LDC2012T16）
2. GALE中英词对齐与标注训练第二部分——新闻专线（LDC2012T20）
3. GALE中英词对齐与标注训练第三部分——网络数据（LDC2012T24）
4. GALE中英词对齐与标注训练第四部分——网络数据（LDC2013T05）
5. GALE中英词对齐与标注——广播训练第一部分（LDC2013T23）
6. GALE阿英词对齐训练第一部分——新闻专线与网络（LDC2014T05）
7. GALE阿英词对齐训练第二部分——新闻专线（LDC2014T10）
8. GALE阿英词对齐训练第三部分——网络数据（LDC2014T14）
9. GALE阿英词对齐——广播训练第一部分（LDC2014T19）

### 数据说明
本次发布的语料为LDC于2007年至2009年间采集的阿拉伯语源广播新闻与广播对话数据。各维度统计分布如下：
| 语言 | 体裁 | 文件数 | 词数 | Token数 | 片段数 |
|--------|--------|--------|---------|----------|---------|
| 阿拉伯语 | 广播对话（BC） | 369 | 97,514 | 129,233 | 7,941 |
| 阿拉伯语 | 广播新闻（BN） | 40 | 70,635 | 86,400 | 3,752 |
| **总计** | —— | 409 | 168,149 | 215,923 | 11,693 |
> 注：词数统计基于未分词的阿拉伯语源数据，Token数统计基于已分词的阿拉伯语源数据。
本次阿拉伯语词对齐任务包含以下环节：
1. 按需对已分词Token进行归一化处理
2. 识别不同类型的对齐链接
3. 判定不适用于标注的语句片段
4. 为依附于其他词或短语的未匹配词添加标注

### 示例
请参阅以下样例：英语原文、英语Token化文本、阿拉伯语原文、阿拉伯语Token化文本、词对齐结果

### 资助说明
本研究部分受美国国防高级研究计划局GALE计划资助（资助编号：HR0011-06-1-0003）。本文内容不一定代表美国政府的立场或政策，不应推断存在官方背书。

### 更新说明
暂无近期更新计划。

### 版权声明
部分内容© 2008-2009 Al Arabiyah、© 2008 Abu Dhabi TV、Al Alam News Channel、Al Baghdadya TV、Al Hiwar、Aljazeera、Al Sharqiya、Dubai TV、Nile TV、Oman TV、PAC Ltd、Saudi TV、Syria TV；© 2008-2009、2014 Trustees of the University of Pennsylvania

搜集汇总

背景与挑战

背景概述

该数据集是GALE项目下用于机器翻译训练的阿拉伯语-英语词对齐数据，包含21.6万个标记的广播新闻和对话平行文本，并带有语言学标签以提升对齐和翻译质量。它是DARPA GALE计划训练数据的一部分，适用于自动内容提取、检索和翻译等自然语言处理任务。