GALE Arabic-English Parallel Aligned Treebank -- Newswire

<h3>Introduction</h3> <p>GALE Arabic-English Parallel Aligned Treebank -- Newswire was developed by the Linguistic Data Consortium (LDC) and contains 267,520 tokens of word aligned Arabic and English parallel text with treebank annotations. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.</p> <p>Parallel aligned treebanks are treebanks annotated with morphological and syntactic structures aligned at the sentence level and the sub-sentence level. Such data sets are useful for natural language processing and related fields, including automatic word alignment system training and evaluation, transfer-rule extraction, word sense disambiguation, translation lexicon extraction and cultural heritage and cross-linguistic studies. With respect to machine translation system development, parallel aligned treebanks may improve system performance with enhanced syntactic parsers, better rules and knowledge about language pairs and reduced word error rate.</p> <p> In this release, the source Arabic data was translated into English. Arabic and English treebank annotations were performed independently. The parallel texts were then word aligned. The material in this corpus corresponds to the Arabic treebanked data appearing in Arabic Treebank: Part 3 v 3.2 (<a href="http://catalog.ldc.upenn.edu/LDC2010T08" rel="nofollow">LDC2010T08</a>) (ATB) and to the English treebanked data in English Translation Treebank: An-Nahar Newswire (<a href="http://catalog.ldc.upenn.edu/LDC2012T02" rel="nofollow">LDC2012T02</a>).</p> <h3>Data</h3> <p>The source data consists of Arabic newswire from the Lebanese publication An Nahar collected by LDC in 2002. All data is encoded as UTF-8. A count of files, words, tokens and segments is below.</p> <table> <tr> <td>Language</td> <td>Files</td> <td>Words</td> <td>Tokens</td> <td>Segments</td> </tr> <tr> <td>Arabic</td> <td>364</td> <td>182,351</td> <td>267,520</td> <td>7,711</td> </tr> </table><p>Note: Word count is based on the untokenized Arabic source and token count is based on the ATB-tokenized Arabic source.</p> <p>The purpose of the GALE word alignment task was to find correspondences between words, phrases or groups of words in a set of parallel texts. Arabic-English word alignment annotation consisted of the following tasks:</p> <ul> <li>Identifying different types of links: translated (correct or incorrect) and not translated (correct or incorrect)</li> <li>Identifying sentence segments not suitable for annotation, e.g., blank segments, incorrectly-segmented segments, segments with foreign languages</li> <li>Tagging unmatched words attached to other words or phrases</li> </ul><p>This release contains four types of files - raw, tokenized, treebank, and wa. The raw format contains the original Arabic and English sentences without any annotation. The tokenized format is the treebank tokenized version of the raw data. It may contain <i>Empty Category</i> tokens (treebank leaves that have the POS label -NONE-). The treebank and wa files are treebank and word alignment annotations on the tokenized files.</p> <h3>Samples</h3> <p>Please view the following samples:</p> <ul> <li><a href="./desc/addenda/LDC2013T10.arb.raw.jpg" rel="nofollow">Arabic Raw</a></li> <li><a href="./desc/addenda/LDC2013T10.arb.tkn.jpg" rel="nofollow">Arabic Token</a></li> <li><a href="./desc/addenda/LDC2013T10.arb.tree.txt" rel="nofollow">Arabic Treebank</a></li> <li><a href="./desc/addenda/LDC2013T10.eng.raw.txt" rel="nofollow">English Raw</a></li> <li><a href="./desc/addenda/LDC2013T10.eng.tkn.txt" rel="nofollow">English Token</a></li> <li><a href="./desc/addenda/LDC2013T10.eng.tree.txt" rel="nofollow">English Treebank</a></li> <li><a href="./desc/addenda/LDC2013T10.wa.txt" rel="nofollow">Word Alignment</a></li> </ul><h3>Sponsorship</h3> <p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.</p> <h3>Updates</h3> <p> None at this time. </p> </br>
Portions © 2002 An Nahar, © 2002-2013 Trustees of the University of Pennsylvania

<h3>引言</h3> <p>GALE阿英平行对齐树库——新闻专线语料库由语言数据联盟（Linguistic Data Consortium，LDC）开发，包含267520个经过词对齐处理的阿拉伯语与英语平行文本，并附带树库标注。该语料曾作为美国国防高级研究计划局（Defense Advanced Research Projects Agency，DARPA）主导的GALE（Global Autonomous Language Exploitation，全球自主语言开发）项目的训练数据。</p> <p>平行对齐树库是在句子级与分句级进行对齐，并附带形态与句法结构标注的树库。此类语料集可应用于自然语言处理及相关领域，包括自动词对齐系统的训练与评估、迁移规则抽取、词义消歧、翻译词典抽取，以及文化遗产与跨语言研究。针对机器翻译系统研发，平行对齐树库可通过优化句法解析器、完善语言对规则与知识库、降低词错误率，进而提升系统性能。</p> <p>本版本的源语为阿拉伯语，经翻译转为英语。阿拉伯语与英语的树库标注均独立完成，随后对平行文本进行词对齐处理。本语料库中的语料分别对应《阿拉伯语树库：第3部分 v3.2》（<a href="http://catalog.ldc.upenn.edu/LDC2010T08" rel="nofollow">LDC2010T08</a>，简称ATB）中的阿拉伯语树库语料，以及《英语翻译树库：An-Nahar新闻专线》（<a href="http://catalog.ldc.upenn.edu/LDC2012T02" rel="nofollow">LDC2012T02</a>）中的英语树库语料。</p> <h3>数据</h3> <p>源数据为LDC于2002年采集自黎巴嫩报刊《An Nahar》的阿拉伯语新闻语料，所有数据均以UTF-8编码。以下为文件数、词数、词元数与分段数的统计：</p> <table> <tr> <td>语言</td> <td>文件数</td> <td>词数</td> <td>词元数</td> <td>分段数</td> </tr> <tr> <td>阿拉伯语</td> <td>364</td> <td>182,351</td> <td>267,520</td> <td>7,711</td> </tr> </table><p>注：词数基于未分词的阿拉伯语源语，词元数基于经过ATB分词的阿拉伯语源语。</p> <p>GALE词对齐任务的目标是找出平行文本集合中单词、短语或词组之间的对应关系。阿英词对齐标注包含以下任务：</p> <ul> <li>识别不同类型的对齐链接：可翻译（正确或错误）与不可翻译（正确或错误）</li> <li>识别不适用于标注的句子分段，例如空白分段、错误分词的分段、包含外语的分段</li> <li>对附着于其他单词或短语的未匹配单词进行标记</li> </ul><p>本版本包含四类文件：原始文件、分词文件、树库文件与词对齐文件。原始格式包含未经过任何标注的阿拉伯语与英语原句。分词格式为原始数据的树库分词版本，可能包含空范畴（Empty Category）词元（即词性标签为-NONE-的树库叶节点）。树库文件与词对齐文件分别为对分词文件所做的树库标注与词对齐标注。</p> <h3>示例</h3> <p>请查看以下示例：</p> <ul> <li><a href="./desc/addenda/LDC2013T10.arb.raw.jpg" rel="nofollow">阿拉伯语原始文本</a></li> <li><a href="./desc/addenda/LDC2013T10.arb.tkn.jpg" rel="nofollow">阿拉伯语分词文本</a></li> <li><a href="./desc/addenda/LDC2013T10.arb.tree.txt" rel="nofollow">阿拉伯语树库文本</a></li> <li><a href="./desc/addenda/LDC2013T10.eng.raw.txt" rel="nofollow">英语原始文本</a></li> <li><a href="./desc/addenda/LDC2013T10.eng.tkn.txt" rel="nofollow">英语分词文本</a></li> <li><a href="./desc/addenda/LDC2013T10.eng.tree.txt" rel="nofollow">英语树库文本</a></li> <li><a href="./desc/addenda/LDC2013T10.wa.txt" rel="nofollow">词对齐结果</a></li> </ul><h3>资助说明</h3> <p>本研究部分由美国国防高级研究计划局GALE项目资助，项目编号HR0011-06-1-0003。本出版物内容不一定反映政府的立场或政策，不应视为获得官方背书。</p> <h3>更新情况</h3> <p>暂无最新更新。</p> <br/> <p>部分内容 © 2002 An Nahar，© 2002-2013 宾夕法尼亚大学理事会</p>

搜集汇总

背景与挑战

背景概述

该数据集是GALE项目的一部分，包含267,520个词符的阿拉伯语-英语平行新闻专线文本，具有树库和词对齐注释，用于机器翻译等自然语言处理任务。数据来源于2002年黎巴嫩An Nahar的阿拉伯语新闻，经翻译和独立标注后形成对齐资源，支持跨语言研究和系统开发。