NIST 2002 Open Machine Translation (OpenMT) Evaluation

<h3>Introduction</h3><br>
<p>NIST 2002 Open Machine Translation (OpenMT) Evaluation is a package containing source data, reference translations, and scoring software used in the NIST 2002 OpenMT evaluation. It is designed to help evaluate the effectiveness of machine translation systems. The package was compiled and scoring software was developed by researchers at NIST, making use of newswire source data and reference translations collected and developed by LDC.</p><br>
<p>The objective of the NIST OpenMT evaluation series is to support research in, and help advance the state of the art of, machine translation (MT) technologies -- technologies that translate text between human languages. Input may include all forms of text. The goal is for the output to be an adequate and fluent translation of the original.</p><br>
<p>The MT evaluation series started in 2001 as part of the DARPA TIDES (Translingual Information Detection, Extraction) program. Beginning with the 2006 evaluation, the evaluations have been driven and coordinated by NIST as NIST OpenMT. These evaluations provide an important contribution to the direction of research efforts and the calibration of technical capabilities in MT. The OpenMT evaluations are intended to be of interest to all researchers working on the general problem of automatic translation between human languages. To this end, they are designed to be simple, to focus on core technology issues, and to be fully supported. The 2002 task was to evaluate translation from Chinese to English and from Arabic to English.</p><br>
<p>Additional information about these evaluations may be found at the <a href="https://www.nist.gov/itl/iad/mig/open-machine-translation-evaluation" rel="nofollow">NIST Open Machine Translation (OpenMT) Evaluation web site</a>.</p><br>
<h3>Scoring Tools</h3><br>
<p>This evaluation kit includes a single perl script (mteval-v09.pl) that may be used to produce a translation quality score for one (or more) MT systems. The script works by comparing the system output translation with a set of (expert) reference translations of the same source text. Comparison is based on finding sequences of words in the reference translations that match word sequences in the system output translation. More information on the evaluation algorithm may be obtained from the paper detailing the algorithm: <a href="http://www.aclweb.org/anthology/P/P02/P02-1040.pdf" rel="nofollow">BLEU: a Method for Automatic Evaluation of Machine Translation (Papineni et al, 2002)</a>.</p><br>
<h3>Data</h3><br>
<p>The Chinese-language source text included in this corpus is a reorganization of data that was initially released to the public as <a href="http://catalog.ldc.upenn.edu/LDC2003T17" rel="nofollow">Multiple-Translation Chinese (MTC) Part 2 (LDC2003T17)</a>. The Chinese-language reference translations are a reorganized subset of data from the same MTC corpus. The Arabic-language data (source text and reference translations) is a reorganized subset of data that was initially released to the public as <a href="http://catalog.ldc.upenn.edu/LDC2003T18" rel="nofollow">Multiple-Translation Arabic (MTA) Part 1 (LDC2003T18)</a>. All source data for this corpus is newswire text. Chinese source text was drawn in March and April 2002 from Xinhua News Agency and in March 2002 from Zaobao News Service (sources indicated in docids). Arabic source text was drawn from the Xinhua News Agency's Arabic newswire feed (October 2001, in the docid range: artb_500 - artb_565) and Agence France-Presse (Feb. 1998 - Oct. 1999, in the docid range: artb_001 - artb_069). Arabic Agence France-Presse source text was also released as part of <a href="http://catalog.ldc.upenn.edu/LDC2001T55" rel="nofollow">Arabic Newswire Part 1 (LDC2001T55)</a>. For details on the methodology of the source data collection and production of reference translations, see the documentation for the above-mentioned corpora.</p><br>
<p>For each language, the test set consists of two files, a source and a reference file. Each reference file contains four independent translations of the data set. The evaluation year, source language, test set (which, by default, is "evalset"), version of the data, and source vs. reference file (with the latter being indicated by "-ref") are reflected in the file name.</p><br>
<p>DARPA TIDES MT and NIST OpenMT evaluations used SGML-formatted test data until 2008 and XML-formatted test data thereafter. The files in this package are provided in both formats.</p><br>
<h3>&nbsp;</h3><br>
<h3>Updates</h3><br>
<p>No updates have been issued at this time.</p></br>
Portions © 1998, 1999 Agence France Presse, © 2002 SPH AsiaOne Ltd, © 2001, 2001 Xinhua News Agency, © 2001, 2003, 2010 Trustees of the University of Pennsylvania

<h3>简介</h3><br><p>美国国家标准与技术研究院（National Institute of Standards and Technology, NIST）2002开放式机器翻译（Open Machine Translation, OpenMT）评估套件包含NIST 2002年OpenMT评估所用的源数据、参考译文与评分软件，旨在助力评估机器翻译系统的效能。该套件由NIST研究人员编译开发评分软件，其源数据与参考译文由语言数据联盟（Linguistic Data Consortium, LDC）采集整理。</p><br><p>OpenMT评估系列的目标是支持机器翻译（Machine Translation, MT）技术的研究，并推动该技术的发展——机器翻译技术指实现不同人类语言间文本互译的技术。评估输入可涵盖各类文本，目标是输出适配且流畅的原文译文。</p><br><p>该机器翻译评估系列始于2001年，作为美国国防高级研究计划局（Defense Advanced Research Projects Agency, DARPA）TIDES（跨语言信息检测、提取）计划的一部分。自2006年评估起，相关评估由NIST主导并协调，正式命名为NIST OpenMT评估。此类评估为机器翻译领域的研究方向与技术能力校准提供了重要支撑。OpenMT评估面向所有致力于跨人类语言自动翻译通用问题的研究人员，因此设计力求简洁，聚焦核心技术议题，并提供全面支持。2002年的评估任务为汉译英与阿译英。</p><br><p>有关此类评估的更多信息，可访问<a href="https://www.nist.gov/itl/iad/mig/open-machine-translation-evaluation" rel="nofollow">NIST开放式机器翻译（OpenMT）评估官网</a>。</p><br><h3>评分工具</h3><br><p>本评估套件包含单个Perl脚本（mteval-v09.pl），可用于生成单个（或多个）机器翻译系统的翻译质量评分。该脚本通过将系统输出译文与同一源文本的多份（专家级）参考译文进行比对来实现评分，比对基于匹配参考译文与系统输出译文的词序列。有关评估算法的更多细节，可参阅详述该算法的论文：<a href="http://www.aclweb.org/anthology/P/P02/P02-1040.pdf" rel="nofollow">《BLEU：一种机器翻译自动评估方法》（Papineni等人，2002年）</a>。</p><br><h3>数据集</h3><br><p>本语料库包含的中文源文本是对最初公开发布的<a href="http://catalog.ldc.upenn.edu/LDC2003T17" rel="nofollow">多译中文（MTC）第二部分（LDC2003T17）</a>数据的重新整理。中文参考译文则来自同一MTC语料库的重新整理子集。阿拉伯语数据（源文本与参考译文）是对最初公开发布的<a href="http://catalog.ldc.upenn.edu/LDC2003T18" rel="nofollow">多译阿拉伯语（MTA）第一部分（LDC2003T18）</a>数据的重新整理子集。本语料库的所有源数据均为新闻专线文本。中文源文本采集自2002年3-4月的新华社新闻，以及2002年3月的《联合早报》通讯社（文档ID中注明了来源）。阿拉伯语源文本采集自新华社阿拉伯语新闻专线（2001年10月，文档ID范围：artb_500 - artb_565）与法新社（1998年2月-1999年10月，文档ID范围：artb_001 - artb_069）。法新社阿拉伯语源文本也曾作为<a href="http://catalog.ldc.upenn.edu/LDC2001T55" rel="nofollow">阿拉伯语新闻专线第一部分（LDC2001T55）</a>的一部分公开发布。有关源数据采集与参考译文制作的方法细节，请参阅上述语料库的官方文档。</p><br><p>针对每种语言，测试集包含两个文件，分别为源文件与参考文件。每份参考文件包含数据集的四份独立译文。文件名可反映评估年份、源语言、测试集（默认值为"evalset"）、数据版本，以及源文件/参考文件（参考文件以"-ref"标识）。</p><br><p>DARPA TIDES MT与NIST OpenMT评估在2008年前使用标准通用标记语言（SGML）格式的测试数据，2008年之后改用可扩展标记语言（XML）格式。本套件中的文件同时提供了两种格式。</p><br><h3>&nbsp;</h3><br><h3>更新说明</h3><br><p>截至目前，尚未发布任何更新。</p><br><p>部分内容 © 1998、1999 法新社，© 2002 新加坡亚洲One有限公司，© 2001、2002 新华社，© 2001、2003、2010 宾夕法尼亚大学托管委员会</p>

搜集汇总

背景与挑战

背景概述

NIST 2002 Open Machine Translation (OpenMT) Evaluation是一个机器翻译评估数据集，发布于2010年，主要用于评估从中文到英文和阿拉伯语到英文的翻译系统效果。它包含新闻通讯社的源数据、多个参考翻译以及基于BLEU算法的评分工具，数据格式支持SGML和XML，旨在支持机器翻译技术的研究与进步。