GALE Arabic-English Word Alignment -- Broadcast Training Part 1

<h3>Introduction</h3><br>
<p>GALE Arabic-English Word Alignment -- Broadcast Training Part 1 was developed by the Linguistic Data Consortium (LDC) and contains 267,257 tokens of word aligned Arabic and English parallel text enriched with linguistic tags. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.</p><br>
<p>Some approaches to statistical machine translation include the incorporation of linguistic knowledge in word aligned text as a means to improve automatic word alignment and machine translation quality. This is accomplished with two annotation schemes: alignment and tagging. Alignment identifies minimum translation units and translation relations by using minimum-match and attachment annotation approaches. A set of word tags and alignment link tags are designed in the tagging scheme to describe these translation units and relations. Tagging adds contextual, syntactic and language-specific features to the alignment annotation.</p><br>
<p>Other releases available in this series are:</p><br>
<ul><br>
<li>GALE Chinese-English Word Alignment and Tagging Training Part 1 -- Newswire and Web (<a href="http://catalog.ldc.upenn.edu/LDC2012T16">LDC2012T16</a>)</li><br>
<li>GALE Chinese-English Word Alignment and Tagging Training Part 2 -- Newswire (<a href="http://catalog.ldc.upenn.edu/LDC2012T20">LDC2012T20</a>)</li><br>
<li>GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web (<a href="http://catalog.ldc.upenn.edu/LDC2012T24">LDC2012T24</a>)</li><br>
<li>GALE Chinese-English Word Alignment and Tagging Training Part 4 -- Web (<a href="http://catalog.ldc.upenn.edu/LDC2013T05">LDC2013T05</a>)</li><br>
<li>GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 1 (<a href="http://catalog.ldc.upenn.edu/LDC2013T23">LDC2013T23</a>)</li><br>
<li>GALE Arabic-English Word Alignment Training Part 1 -- Newswire and Web (<a href="http://catalog.ldc.upenn.edu/LDC2014T05">LDC2014T05</a>)</li><br>
<li>GALE Arabic-English Word Alignment Training Part 2 -- Newswire (<a href="http://catalog.ldc.upenn.edu/LDC2014T10">LDC2014T10</a>)</li><br>
<li>GALE Arabic-English Word Alignment Training Part 3 -- Web (<a href="../../../LDC2014T14">LDC2014T14</a>)</li><br>
</ul><br>
<h3>Data</h3><br>
<p>This release consists of Arabic source broadcast news and broadcast conversation data collected by LDC from 2007-2009. The distribution by genre, words, tokens and segments appears below:</p><br>
<p>&nbsp;</p><br>
<table border="1"><br>
<tbody><br>
<tr><th>Language</th><th>Genre</th><th>Files</th><th>Words</th><th>Tokens</th><th>Segments</th></tr><br>
<tr><br>
<td>Arabic</td><br>
<td>BC</td><br>
<td>231</td><br>
<td>79,485</td><br>
<td>103,816</td><br>
<td>4,114</td><br>
</tr><br>
<tr><br>
<td>Arabic</td><br>
<td>BN</td><br>
<td>92</td><br>
<td>131,789</td><br>
<td>163,441</td><br>
<td>7,227</td><br>
</tr><br>
<tr><br>
<td>Totals</td><br>
<td>&nbsp;</td><br>
<td>323</td><br>
<td>211,274</td><br>
<td>267,257</td><br>
<td>11,341</td><br>
</tr><br>
</tbody><br>
</table><br>
<p>&nbsp;</p><br>
<p>Note that word count is based on the untokenized Arabic source, and token count is based on the tokenized Arabic source.</p><br>
<p>The Arabic word alignment tasks consisted of the following components:</p><br>
<ul><br>
<li>Normalizing tokenized tokens as needed</li><br>
<li>Identifying different types of links</li><br>
<li>Identifying sentence segments not suitable for annotation</li><br>
<li>Tagging unmatched words attached to other words or phrases</li><br>
</ul><br>
<h3>Samples</h3><br>
<p>Please view the following samlpes:</p><br>
<ul><br>
<li><a href="desc/addenda/LDC2014T19.eng.raw.txt">English Raw</a></li><br>
<li><a href="desc/addenda/LDC2014T19.eng.tkn.txt">English Token</a></li><br>
<li><a href="desc/addenda/LDC2014T19.arb.raw.jpg">Arabic Raw</a></li><br>
<li><a href="desc/addenda/LDC2014T19.arb.tkn.jpg">Arabic Token</a></li><br>
<li><a href="desc/addenda/LDC2014T19.wa.txt">Word Alignment</a></li><br>
</ul><br>
<h3>Sponsorship</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Updates</h3><br>
<p>None at this time.</p></br>
Portions © 2008 Abu Dhabi TV, © 2008 Al Alam News Channel, © 2008-2009 Al Arabiyah, © 2008 Al Baghdadya TV, © 2008-2009 Al Fayha, © 2008-2009 Al Hiwar, © 2008 Al Iraqiyah, © 2008-2009 Al Ordiniyah, © 2008 Bahrain TV, © 2008-2009 Dubai TV, © 2008 Nile TV, © 2008 Oman TV, © 2008 PAC Ltd, © 2008-2009 Saudi TV, © 2008 Syria TV, © 2008-2009 Tunisian National Television, © 2008 Yemen TV, © 2007-2009, 2014 Trustees of the University of Pennsylvania

<h3>引言</h3><br><p>GALE阿语-英语词对齐——广播训练集第1部分由语言数据联盟（Linguistic Data Consortium，LDC）开发，包含267,257个Token（Token）的带语言标记的阿语-英语平行对齐文本。该数据集曾作为训练数据应用于美国国防高级研究计划局（Defense Advanced Research Projects Agency，DARPA）的全球自主语言开发（Global Autonomous Language Exploitation，GALE）项目。</p><br><p>部分统计机器翻译方法会将语言知识融入词对齐文本中，以此提升自动词对齐效果与机器翻译质量。这一目标可通过两类标注方案实现：对齐标注与标记标注。对齐标注采用最小匹配与依附标注的方式，识别最小翻译单元及翻译关联关系。标记标注方案中设计了一套词标记与对齐链接标记，用于描述上述翻译单元与关联关系，为对齐标注添加上下文、句法及语言专属特征。</p><br><p>本系列其他已发布数据集包括：</p><br><ul><br><li>GALE汉英词对齐与标记训练集第1部分——新闻专线与网络文本（<a href="http://catalog.ldc.upenn.edu/LDC2012T16">LDC2012T16</a>）</li><br><li>GALE汉英词对齐与标记训练集第2部分——新闻专线（<a href="http://catalog.ldc.upenn.edu/LDC2012T20">LDC2012T20</a>）</li><br><li>GALE汉英词对齐与标记训练集第3部分——网络文本（<a href="http://catalog.ldc.upenn.edu/LDC2012T24">LDC2012T24</a>）</li><br><li>GALE汉英词对齐与标记训练集第4部分——网络文本（<a href="http://catalog.ldc.upenn.edu/LDC2013T05">LDC2013T05</a>）</li><br><li>GALE汉英词对齐与标记训练集——广播训练集第1部分（<a href="http://catalog.ldc.upenn.edu/LDC2013T23">LDC2013T23</a>）</li><br><li>GALE阿语-英语词对齐训练集第1部分——新闻专线与网络文本（<a href="http://catalog.ldc.upenn.edu/LDC2014T05">LDC2014T05</a>）</li><br><li>GALE阿语-英语词对齐训练集第2部分——新闻专线（<a href="http://catalog.ldc.upenn.edu/LDC2014T10">LDC2014T10</a>）</li><br><li>GALE阿语-英语词对齐训练集第3部分——网络文本（<a href="../../../LDC2014T14">LDC2014T14</a>）</li><br></ul><br><h3>数据</h3><br><p>本发布数据集包含LDC于2007年至2009年采集的阿语源广播新闻与广播对话数据。各类别下的体裁、文件数、词数、Token数及分段数如下：</p><br><p>&nbsp;</p><br><table border="1"><br><tbody><br><tr><th>语言</th><th>体裁</th><th>文件数</th><th>词数</th><th>Token数</th><th>分段数</th></tr><br><tr><br><td>阿拉伯语</td><br><td>BC</td><br><td>231</td><br><td>79,485</td><br><td>103,816</td><br><td>4,114</td><br></tr><br><tr><br><td>阿拉伯语</td><br><td>BN</td><br><td>92</td><br><td>131,789</td><br><td>163,441</td><br><td>7,227</td><br></tr><br><tr><br><td>总计</td><br><td>&nbsp;</td><br><td>323</td><br><td>211,274</td><br><td>267,257</td><br><td>11,341</td><br></tr><br></tbody><br></table><br><p>&nbsp;</p><br><p>请注意，词数基于未分词的阿语源文本统计，Token数则基于分词后的阿语源文本统计。</p><br><p>阿语词对齐任务包含以下环节：</p><br><ul><br><li>按需对分词后的Token进行归一化处理</li><br><li>识别不同类型的对齐链接</li><br><li>标记不适用于标注的句子分段</li><br><li>为依附于其他词或短语的未匹配词添加标记</li><br></ul><br><h3>示例</h3><br><p>请查看以下示例：</p><br><ul><br><li><a href="desc/addenda/LDC2014T19.eng.raw.txt">英语原文</a></li><br><li><a href="desc/addenda/LDC2014T19.eng.tkn.txt">英语分词文本</a></li><br><li><a href="desc/addenda/LDC2014T19.arb.raw.jpg">阿语原文</a></li><br><li><a href="desc/addenda/LDC2014T19.arb.tkn.jpg">阿语分词文本</a></li><br><li><a href="desc/addenda/LDC2014T19.wa.txt">词对齐结果</a></li><br></ul><br><h3>资助信息</h3><br><p>本研究部分由美国国防高级研究计划局GALE项目资助，资助编号为HR0011-06-1-0003。本文内容不一定代表政府的立场或政策，不应被视为存在官方背书。</p><br><h3>更新情况</h3><br><p>暂无更新。</p><br><p>部分内容 © 2008 阿布扎比电视台、© 2008 阿拉姆新闻频道、© 2008-2009 阿拉伯电视台、© 2008 巴格达迪亚电视台、© 2008-2009 法哈电视台、© 2008-2009 希瓦尔电视台、© 2008 伊拉克迪亚电视台、© 2008 奥迪尼亚电视台、© 2008 巴林电视台、© 2008-2009 迪拜电视台、© 2008 尼罗河电视台、© 2008 阿曼电视台、© 2008 PAC有限公司、© 2008-2009 沙特阿拉伯电视台、© 2008 叙利亚电视台、© 2008-2009 突尼斯国家电视台、© 2008 也门电视台、© 2007-2009、2014 宾夕法尼亚大学托管委员会</p>

搜集汇总

背景与挑战

背景概述

该数据集是GALE项目中的阿拉伯语-英语词对齐训练数据第一部分，包含267,257个词对齐的tokens，来源于2007-2009年的广播新闻和广播对话，用于支持机器翻译和语言处理任务。数据集通过标注方案（对齐和标签）增强了语言特征，旨在提高自动词对齐和机器翻译质量。