ISI Arabic-English Automatically Extracted Parallel Text

<p>This distribution contains a corpus of Arabic-English parallel sentences, which were extracted automatically from two monolingual corpora: Arabic Gigaword Second Edition (LDC2006T02) and English Gigaword Second Edition (LDC2005T12). The data was extracted from news articles published by Xinhua News Agency and Agence France Presse and was obtained using the automatic parallel sentence identification method described in the following publication: Dragos Stefan Munteanu, Daniel Marcu, 2005. Machine Translation Performance by Exploiting Non-parallel Corpora, Computational Linguistics, 31(4):477-504</p><br>
<p>The corpus contains 1,124,609 sentence pairs; the word count on the English side is approximately 31M words. The sentences in the parallel corpus preserve the form and encoding of the texts in the original Gigaword corpora.</p><br>
<p>For each sentence pair in the corpus the authors provide the names of the documents from which the two sentences were extracted, as well as a confidence score (between 0.5 and 1.0), which is indicative of their degree of parallelism. The parallel sentence identification approach is designed to judge sentence pairs in isolation from their contexts, and can therefore find parallel sentences within document pairs which are not parallel. The fact that two documents share several parallel sentences does not necessarily mean the documents are parallel.</p><br>
<p>In order to make this resource useful for research in Machine Translation (MT), the authors made efforts to detect potential overlaps between this data and the standard test and development data sets used by the MT community. The NIST 2002-2005 MT evaluation data sets contain several articles from Xinhua News Agency and Agence France Presse. Sentence pairs in this distribution that have a 7-gram overlap with a sentence pair in a NIST MT evaluation set or sentence pairs coming from documents whose names are similar to those in the NIST MT sets are marked with a negative confidence score.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data in this publication, please examine this <a href="desc/addenda/LCD2007T08_para.jpg" rel="nofollow">image of text data</a>.</p></br>
Portions © 1994-2004 Agence France Presse, © 1995-2004 Xinhua News Agency, © 2005, 2006, 2007 Trustees of the University of Pennsylvania

<p>本数据集发布包包含阿英平行语句语料，该语料自动提取自两份单语语料库：阿拉伯语Gigaword第二版（Arabic Gigaword Second Edition，LDC2006T02）与英语Gigaword第二版（English Gigaword Second Edition，LDC2005T12）。本次提取的数据源自新华社（Xinhua News Agency）与法新社（Agence France Presse）发布的新闻稿件，提取方法采用下述文献中提出的自动平行语句识别方案：Dragos Stefan Munteanu、Daniel Marcu，2005年。《利用非平行语料提升机器翻译性能》，《计算语言学》，31(4):477-504。</p><br>
<p>该语料库共计1,124,609条语句对，英语侧词量约为3100万。平行语料中的语句完整保留了原始Gigaword语料库文本的格式与编码方式。</p><br>
<p>针对语料库中的每条语句对，作者提供了两条语句各自来源的文档名称，以及一条置信度评分（取值范围0.5至1.0），该评分可反映语句对的平行程度。本次采用的平行语句识别方案仅独立判定语句对，无需依赖上下文，因此可在非平行的文档对中识别出平行语句。两份文档存在多条平行语句，并不代表这两份文档本身为平行文档。</p><br>
<p>为使该资源可服务于机器翻译（Machine Translation，MT）领域研究，作者已尝试排查本数据集与机器翻译社区通用的标准测试、开发数据集间的潜在重复内容。美国国家标准与技术研究院（National Institute of Standards and Technology，NIST）2002-2005机器翻译评测数据集包含多篇来自新华社与法新社的稿件。本发布包中，若某语句对与NIST机器翻译评测集内的语句对存在7元组（7-gram）重叠，或其来源文档名称与NIST机器翻译评测集内的文档名称相近，则将该语句对标记为负置信度评分。</p><br>
<h3>示例样本</h3><br>
<p>若需查看本数据集的示例数据，请参阅该<a href="desc/addenda/LCD2007T08_para.jpg" rel="nofollow">文本数据图片</a>。</p></br>
部分内容 © 1994-2004 法新社（Agence France Presse）、© 1995-2004 新华社（Xinhua News Agency）、© 2005、2006、2007 宾夕法尼亚大学理事会（Trustees of the University of Pennsylvania）

搜集汇总

背景与挑战

背景概述

ISI Arabic-English Automatically Extracted Parallel Text 是一个从阿拉伯语和英语Gigaword单语语料库中自动提取的平行句子语料库，包含1,124,609个句子对，英语侧单词数约3100万。该数据集专为机器翻译研究设计，每个句子对都提供置信度分数以评估并行程度，并标记了与标准测试集的重叠部分，确保数据实用性。