English Gigaword Fifth Edition

<h3>Introduction</h3><br>
<p>English Gigaword Fifth Edition is a comprehensive archive of newswire text data that has been acquired over several years by the Linguistic Data Consortiume (LDC). The fifth edition includes all of the contents in English Gigaword Fourth Edition (<a href="http://catalog.ldc.upenn.edu/LDC2009T13" rel="nofollow">LDC2009T13</a>) plus new data covering the 24-month period January 2009 through December 2010.</p><br>
<p>The seven distinct international sources of English newswire included in this edition are the following:</p><br>
<ul><br>
<li>Agence France-Presse, English Service (afp_eng)</li><br>
<li>Associated Press Worldstream, English Service (apw_eng)</li><br>
<li>Central News Agency of Taiwan, English Service (cna_eng)</li><br>
<li>Los Angeles Times/Washington Post Newswire Service (ltw_eng)</li><br>
<li>Washington Post/Bloomberg Newswire Service (wpb_eng)</li><br>
<li>New York Times Newswire Service (nyt_eng)</li><br>
<li>Xinhua News Agency, English Service (xin_eng)</li><br>
</ul><br>
<p>The seven letter codes in the parentheses above include the three-character source name abbreviations and the three-character langauge code (eng) separated by an underscore (_) character. The three-letter language code conforms to LDC's internal convention based on the <a href="http://www.sil.org/iso639-3/" rel="nofollow">ISO 639-3</a> standard.</p><br>
<h3>Data</h3><br>
<p>The following table sets forth the overall totals for each source. Note that Total-MB refers to the quantity of date when unzipped (approximately 26 gigabytes), Gzip-MB refers to compressed file sizes as stored on the DVD-ROMs and K-wrds refers to the number of whitespace-separated tokens (of all types) after all SGML tags are eliminated:</p><br>
<table><br>
<tbody><br>
<tr><br>
<td>Source</td><br>
<td>#Files</td><br>
<td>Gzip-MB</td><br>
<td>Totl-MB</td><br>
<td>K-wrds</td><br>
<td>#DOCs</td><br>
</tr><br>
<tr><br>
<td>afp_eng</td><br>
<td>146</td><br>
<td>1732</td><br>
<td>4937</td><br>
<td>738322</td><br>
<td>2479624</td><br>
</tr><br>
<tr><br>
<td>apw_eng</td><br>
<td>193</td><br>
<td>2700</td><br>
<td>7889</td><br>
<td>1186955</td><br>
<td>3107777</td><br>
</tr><br>
<tr><br>
<td>cna_eng</td><br>
<td>144</td><br>
<td>86</td><br>
<td>261</td><br>
<td>38491</td><br>
<td>145317</td><br>
</tr><br>
<tr><br>
<td>ltw_eng</td><br>
<td>127</td><br>
<td>651</td><br>
<td>1694</td><br>
<td>268088</td><br>
<td>411032</td><br>
</tr><br>
<tr><br>
<td>nyt_eng</td><br>
<td>197</td><br>
<td>3280</td><br>
<td>8938</td><br>
<td>1422670</td><br>
<td>1962178</td><br>
</tr><br>
<tr><br>
<td>wpb_eng</td><br>
<td>12</td><br>
<td>42</td><br>
<td>111</td><br>
<td>17462</td><br>
<td>26143</td><br>
</tr><br>
<tr><br>
<td>xin_eng</td><br>
<td>191</td><br>
<td>834</td><br>
<td>2518</td><br>
<td>360714</td><br>
<td>1744025</td><br>
</tr><br>
<tr><br>
<td>TOTAL</td><br>
<td>1010</td><br>
<td>9325</td><br>
<td>26348</td><br>
<td>4032686</td><br>
<td>9876086</td><br>
</tr><br>
</tbody><br>
</table><br>
<h3>Sponsorship</h3><br>
<p>This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or policy of the Government, and no official endorsement should be inferred.</p><br>
<h3>Samples</h3><br>
<p>For an example of the data in this corpus, please review this <a href="desc/addenda/LDC2011T07.jpg">sample</a>.</p></br>
Portions © 1994-2010 Agence France Presse, © 1994-2010 The Associated Press, © 1997-2010 Central News Agency (Taiwan), © 1994-1998, 2003-2009 Los Angeles Times-Washington Post News Service, Inc., © 1994-2010 New York Times, © 2010 The Washington Post News Service with Bloomberg News, © 1995-2010 Xinhua News Agency, © 2003, 2005, 2007, 2009, 2011 Trustees of the University of Pennsylvania

搜集汇总

背景与挑战

背景概述

English Gigaword Fifth Edition是一个包含多个国际新闻机构英文新闻电讯的大型文本数据集，总数据量约26GB，涵盖400万单词和近1000万篇文档，适用于语言建模和信息检索等应用。