Arabic Gigaword Fifth Edition

<h3>Introduction</h3> <p>Arabic Gigaword Fifth Edition, Linguistic Data Consortium (LDC) catalog number LDC2011T11 and ISBN 1-58563-595-2, was produced by LDC. It is a comprehensive archive of newswire text data that has been acquired from Arabic news sources by LDC at the University of Pennsylvania. Arabic Gigaword Fifth Edition includes all of the content of the fourth edition of Arabic Gigaword (<a href="http://catalog.ldc.upenn.edu/LDC2009T30" rel="nofollow">LDC2009T30</a>) plus new data covering the period from January 2009 through December 2010.</p> <p>Nine distinct sources of Arabic newswire are represented here:</p> <ul> <li>Asharq Al-Awsat (aaw_arb)</li> <li>Agence France Presse (afp_arb)</li> <li>Al-Ahram (ahr_arb)</li> <li>Assabah (asb_arb)</li> <li>Al Hayat (hyt_arb)</li> <li>An Nahar (nhr_arb)</li> <li>Al-Quds Al-Arabi (qds_arb)</li> <li>Ummah Press (umh_arb)</li> <li>Xinhua News Agency (xin_arb)</li> </ul><p>The seven-character codes shown above represent both the directory names where the data files are found, and the 7-letter prefix that appears at the beginning of every file name. The 7-letter codes consist of the three-character source name IDs and the three-character language code (arb) separated by an underscore (_) character. The three-character language code conforms to the <a href="http://www.sil.org/iso639-3/default.asp" rel="nofollow">ISO 639-3</a> standard.</p> <p>In addition to adding new data, the following updates were made:</p> <ul> <li>Repeated documents in Asharq Al-Awsat data from 2008 were removed.</li> <li>Document formatting and <em>docid</em> duplication problems were corrected in Agence France Presse (AFP) data.</li> <li>Significant duplication of content in 2007-2008 An Nahar data was detected, and the duplicated documents were removed.</li> </ul><p>More details about these changes can be found in the included <a href="./docs/LDC2011T11/0readme.txt" rel="nofollow">readme file</a>.</p> <h3>Data</h3> <p>All text data are presented in SGML form, using a very simple, minimal markup structure. For every opening tag (DOC, HEADLINE, DATELINE, TEXT, P), there is a corresponding closing tag -- always. The attribute values in the DOC tag are always presented within double-quotes the id= attribute of DOC consists of the 7-letter source abbreviation (in CAPS), an underscore character, an 8-digit date string representing the date of the story (YYYYMMDD), a period, and a 4-digit sequence number starting at 0001 for each date (e.g. XIN_ARB_200101.0001) in this way, every DOC in the corpus is uniquely identifiable by the id string.</p> <p>For this release, all sources have received a uniform treatment in terms of quality control, and we have applied a rudimentary (and _approximate_) categorization of DOC units into four distinct types. The classification is indicated by the type=string attribute that is included in each opening DOC tag. The four types are:</p> <ul> <li>story: This is by far the most frequent type, and it represents the most typical newswire item: a coherent report on a particular topic or event, consisting of paragraphs and full sentences.</li> <li>multi: This type of DOC contains a series of unrelated blurbs, each of which briefly describes a particular topic or event this is typically applied to DOCs that contain summaries of todays news, news briefs in ... (some general area like finance or sports), and so on.</li> <li> other: This represents DOCs that clearly do not fall into any of the above types -- in general, items of this type are intended for broad circulation (they are not advisories), they may be topically coherent (unlike multi type DOCs), and they typically do not contain paragraphs or sentences (they arent really stories) these are things like lists of sports scores, stock prices, temperatures around the world, and so on.</li> </ul><p>Other Gigaword corpora (e.g., in English and Chinese) have a fourth category, advis (for advisory), which applies to DOCs that contain text intended solely for news service editors, not the news-reading public. The task of determining patterns for assigning non-story type labels was carried out by a native speaker of Arabic, and the advis category was determined to be inapplicable to the data.</p> <p>Note that the markup was applied algorithmically, using logic that was based on less-than-complete knowledge of the data. For the most part, the HEADLINE, DATELINE and TEXT tags have their intended content but due to the inherent variability (and the inevitable source errors) in the data, users may find occasional mishaps where the headline and/or dateline were not successfully identified (hence show up within TEXT), or where an initial sentence or paragraph has been mistakenly tagged as the headline or dateline.</p> <h3>Sample</h3> <p>Please view this <a href="./desc/addenda/LDC2011T11.jpg" rel="nofollow">sample</a>.</p> <h3>Sponsorship</h3> <p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily refelct the position or policy of the Government, and no official endorsement should be inferred.</p> <h3>Updates</h3> <p>None at this time. </p> </br>
Portions © 1994-2010 Agence France Presse, © 2006-2010 Al-Ahram, © 2006-2010 Al-Quds Al-Arabi, © 2006-2010 Asharq Al-Awsat, © 2004-2010 Assabah, © 1994-2003, 2005-2010 Al Hayat, © 1995-2010 An Nahar, © 2003-2010 Ummah Press, © 2001-2010 Xinhua News Agency, © 2003, 2006, 2007, 2009, 2011 Trustees of the University of Pennsylvania

<h3>引言</h3> <p>阿拉伯语巨文第五版（Arabic Gigaword Fifth Edition）由语言数据联盟（Linguistic Data Consortium，LDC）制作，其馆藏编号为LDC2011T11，国际标准书号为ISBN 1-58563-595-2。该数据集是宾夕法尼亚大学LDC从阿拉伯语新闻来源采集的综合性新闻文本档案。阿拉伯语巨文第五版包含第四版阿拉伯语巨文（<a href="http://catalog.ldc.upenn.edu/LDC2009T30" rel="nofollow">LDC2009T30</a>）的全部内容，以及2009年1月至2010年12月期间的新增数据。</p> <p>本数据集涵盖9个不同的阿拉伯语新闻来源：</p> <ul> <li>《中东日报》（Asharq Al-Awsat，aaw_arb）</li> <li>法新社（Agence France Presse，afp_arb）</li> <li>《金字塔报》（Al-Ahram，ahr_arb）</li> <li>《祖国报》（Assabah，asb_arb）</li> <li>《生活报》（Al Hayat，hyt_arb）</li> <li>《每日新闻报》（An Nahar，nhr_arb）</li> <li>《阿拉伯耶路撒冷报》（Al-Quds Al-Arabi，qds_arb）</li> <li>《民族报》（Ummah Press，umh_arb）</li> <li>新华通讯社（Xinhua News Agency，xin_arb）</li> </ul><p>上述7位代码既是数据文件所在的目录名，也是每个文件名开头的7字母前缀。该7位代码由3位来源标识、下划线（_）及3位语言代码（arb）组成，其中3位语言代码符合<a href="http://www.sil.org/iso639-3/default.asp" rel="nofollow">ISO 639-3</a>标准。</p> <p>除新增数据外，本次发布还进行了以下修正：</p> <ul> <li>移除了2008年《中东日报》数据中的重复文档。</li> <li>修正了法新社（Agence France Presse，AFP）数据中的文档格式与<em>docid</em>重复问题。</li> <li>检测并移除了2007-2008年《每日新闻报》数据中的大量重复内容文档。</li> </ul><p>有关这些修正的更多细节，请参阅附带的<a href="./docs/LDC2011T11/0readme.txt" rel="nofollow">自述文件</a>。</p> <h3>数据说明</h3> <p>所有文本数据均采用标准通用标记语言（SGML）格式，使用极简标记结构。所有起始标记（DOC、HEADLINE、DATELINE、TEXT、P）均配有对应的闭合标记。DOC标签中的属性值始终使用双引号包裹，其中id属性由大写的7位来源缩写、下划线、代表报道日期的8位数字字符串（YYYYMMDD）、句点，以及每日从0001开始的4位序列编号组成（例如XIN_ARB_200101.0001）。通过该格式，语料库中的每篇DOC均可通过id字符串实现唯一标识。</p> <p>本次发布的所有数据源均经过统一的质控流程，同时我们对DOC单元进行了初步（且近似）的分类，将其划分为4种不同类型。分类信息通过每个DOC起始标签中的type=字符串属性标注。四种类型如下：</p> <ul> <li>story：该类型为最常见的新闻条目，代表典型的新闻报道：围绕特定主题或事件展开的连贯报道，包含多个段落与完整句子。</li> <li>multi：该类型DOC包含一系列互不关联的短讯，每条短讯简要介绍某一主题或事件，通常用于包含当日新闻汇总、金融或体育等领域新闻简报的文档。</li> <li>other：该类型代表明显不属于上述三类的文档。这类内容通常面向广泛受众（而非仅供内部参考），可能具备主题连贯性（区别于multi类型文档），但通常不包含段落或完整句子（并非标准新闻报道），例如体育赛事比分列表、股票价格、全球气温数据等。</li> </ul><p>其他语言的巨文语料库（如英语、汉语版）设有第四类advis（即advisory，仅供编辑参考文档），用于包含仅面向新闻编辑而非普通读者的文本的文档。本次非新闻类标签的分类规则由阿拉伯语母语人员制定，经评估后发现advis类别不适用于本数据集。</p> <p>请注意，本次标记通过算法自动完成，其逻辑基于对数据集的不完全认知。绝大多数情况下，HEADLINE、DATELINE与TEXT标签均可正确匹配对应内容，但由于数据本身存在固有差异且不可避免地带有来源错误，用户可能会遇到以下异常情况：标题或发稿地点未能被正确识别（因而出现在TEXT标签内），或首句/首段被错误标注为标题或发稿地点。</p> <h3>示例</h3> <p>请查看该<a href="./desc/addenda/LDC2011T11.jpg" rel="nofollow">示例文件</a>。</p> <h3>资助说明</h3> <p>本项目部分由美国国防高级研究计划局（Defense Advanced Research Projects Agency）GALE计划资助，资助编号为HR0011-06-1-0003。本出版物内容不代表美国政府的立场或政策，亦不代表官方背书。</p> <h3>更新记录</h3> <p>暂无更新记录。</p> <br/> Portions © 1994-2010 法新社（Agence France Presse）、© 2006-2010 《金字塔报》（Al-Ahram）、© 2006-2010 《阿拉伯耶路撒冷报》（Al-Quds Al-Arabi）、© 2006-2010 《中东日报》（Asharq Al-Awsat）、© 2004-2010 《祖国报》（Assabah）、© 1994-2003、2005-2010 《生活报》（Al Hayat）、© 1995-2010 《每日新闻报》（An Nahar）、© 2003-2010 《民族报》（Ummah Press）、© 2001-2010 新华通讯社，© 2003、2006、2007、2009、2011 宾夕法尼亚大学托管委员会。

搜集汇总

背景与挑战

背景概述

Arabic Gigaword Fifth Edition 是一个阿拉伯语新闻文本语料库，由Linguistic Data Consortium于2011年发布，覆盖2009年至2010年的数据，并包含九个新闻来源的文本。该数据集以SGML格式组织，支持自然语言处理、语言建模和信息检索等应用，并进行了文档类型分类以提高数据可用性。