English Gigaword Fifth Edition

Name: English Gigaword Fifth Edition
Creator: Linguistic Data Consortium
Published: 2023-06-06 08:56:32
License: 暂无描述

DataCite Commons2023-06-06 更新2024-07-13 收录

下载链接：

https://catalog.ldc.upenn.edu/LDC2011T07

下载链接

链接失效反馈

官方服务：

资源简介：

<h3>Introduction</h3> English Gigaword Fifth Edition is a comprehensive archive of newswire text data that has been acquired over several years by the Linguistic Data Consortiume (LDC). The fifth edition includes all of the contents in English Gigaword Fourth Edition (<a href="http://catalog.ldc.upenn.edu/LDC2009T13" rel="nofollow">LDC2009T13</a>) plus new data covering the 24-month period January 2009 through December 2010. The seven distinct international sources of English newswire included in this edition are the following: <ul> <li>Agence France-Presse, English Service (afp_eng)</li> <li>Associated Press Worldstream, English Service (apw_eng)</li> <li>Central News Agency of Taiwan, English Service (cna_eng)</li> <li>Los Angeles Times/Washington Post Newswire Service (ltw_eng)</li> <li>Washington Post/Bloomberg Newswire Service (wpb_eng)</li> <li>New York Times Newswire Service (nyt_eng)</li> <li>Xinhua News Agency, English Service (xin_eng)</li> </ul> The seven letter codes in the parentheses above include the three-character source name abbreviations and the three-character langauge code (eng) separated by an underscore (_) character. The three-letter language code conforms to LDC's internal convention based on the <a href="http://www.sil.org/iso639-3/" rel="nofollow">ISO 639-3</a> standard. <h3>Data</h3> The following table sets forth the overall totals for each source. Note that Total-MB refers to the quantity of date when unzipped (approximately 26 gigabytes), Gzip-MB refers to compressed file sizes as stored on the DVD-ROMs and K-wrds refers to the number of whitespace-separated tokens (of all types) after all SGML tags are eliminated: <table> <tbody> <tr> <td>Source</td> <td>#Files</td> <td>Gzip-MB</td> <td>Totl-MB</td> <td>K-wrds</td> <td>#DOCs</td> </tr> <tr> <td>afp_eng</td> <td>146</td> <td>1732</td> <td>4937</td> <td>738322</td> <td>2479624</td> </tr> <tr> <td>apw_eng</td> <td>193</td> <td>2700</td> <td>7889</td> <td>1186955</td> <td>3107777</td> </tr> <tr> <td>cna_eng</td> <td>144</td> <td>86</td> <td>261</td> <td>38491</td> <td>145317</td> </tr> <tr> <td>ltw_eng</td> <td>127</td> <td>651</td> <td>1694</td> <td>268088</td> <td>411032</td> </tr> <tr> <td>nyt_eng</td> <td>197</td> <td>3280</td> <td>8938</td> <td>1422670</td> <td>1962178</td> </tr> <tr> <td>wpb_eng</td> <td>12</td> <td>42</td> <td>111</td> <td>17462</td> <td>26143</td> </tr> <tr> <td>xin_eng</td> <td>191</td> <td>834</td> <td>2518</td> <td>360714</td> <td>1744025</td> </tr> <tr> <td>TOTAL</td> <td>1010</td> <td>9325</td> <td>26348</td> <td>4032686</td> <td>9876086</td> </tr> </tbody> </table> <h3>Sponsorship</h3> This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or policy of the Government, and no official endorsement should be inferred. <h3>Samples</h3> For an example of the data in this corpus, please review this <a href="desc/addenda/LDC2011T07.jpg">sample</a>. Portions © 1994-2010 Agence France Presse, © 1994-2010 The Associated Press, © 1997-2010 Central News Agency (Taiwan), © 1994-1998, 2003-2009 Los Angeles Times-Washington Post News Service, Inc., © 1994-2010 New York Times, © 2010 The Washington Post News Service with Bloomberg News, © 1995-2010 Xinhua News Agency, © 2003, 2005, 2007, 2009, 2011 Trustees of the University of Pennsylvania

<h3>简介</h3> 《英语百万语料库第五版》（English Gigaword Fifth Edition）是由语言数据联盟（Linguistic Data Consortium, LDC）历经数年采集的新闻文本综合存档。该第五版包含了《英语百万语料库第四版》（<a href="http://catalog.ldc.upenn.edu/LDC2009T13" rel="nofollow">LDC2009T13</a>）的全部内容，新增了2009年1月至2010年12月共24个月的新闻数据。 本版收录了7家不同的国际英语新闻来源，具体如下： <ul> <li>法新社英语专线（Agence France-Presse, English Service，缩写afp_eng）</li> <li>美联社全球专线英语服务（Associated Press Worldstream, English Service，缩写apw_eng）</li> <li>台湾中央通讯社英语专线（Central News Agency of Taiwan, English Service，缩写cna_eng）</li> <li>洛杉矶时报/华盛顿邮报新闻专线服务（Los Angeles Times/Washington Post Newswire Service，缩写ltw_eng）</li> <li>华盛顿邮报/彭博新闻专线服务（Washington Post/Bloomberg Newswire Service，缩写wpb_eng）</li> <li>纽约时报新闻专线服务（New York Times Newswire Service，缩写nyt_eng）</li> <li>新华社英语专线（Xinhua News Agency, English Service，缩写xin_eng）</li> </ul> 上述括号内的7位字母编码由3字符来源缩写与3字符语言代码（eng）通过下划线（_）连接而成，其中3字母语言代码遵循基于<a href="http://www.sil.org/iso639-3/" rel="nofollow">ISO 639-3</a>标准的LDC内部规范。 <h3>数据说明</h3> 下表列出了各来源数据的总体统计量。请注意，Totl-MB指解压后的文件总容量（全库解压后约26吉字节），Gzip-MB指存储于DVD-ROM中的压缩文件大小，K-wrds指去除所有SGML标签后按空白符分隔的各类Token（Token）总数。 <table> <tbody> <tr> <td>新闻来源</td> <td>文件数量</td> <td>Gzip-MB</td> <td>Totl-MB</td> <td>K-wrds</td> <td>文档数量</td> </tr> <tr> <td>afp_eng</td> <td>146</td> <td>1732</td> <td>4937</td> <td>738322</td> <td>2479624</td> </tr> <tr> <td>apw_eng</td> <td>193</td> <td>2700</td> <td>7889</td> <td>1186955</td> <td>3107777</td> </tr> <tr> <td>cna_eng</td> <td>144</td> <td>86</td> <td>261</td> <td>38491</td> <td>145317</td> </tr> <tr> <td>ltw_eng</td> <td>127</td> <td>651</td> <td>1694</td> <td>268088</td> <td>411032</td> </tr> <tr> <td>nyt_eng</td> <td>197</td> <td>3280</td> <td>8938</td> <td>1422670</td> <td>1962178</td> </tr> <tr> <td>wpb_eng</td> <td>12</td> <td>42</td> <td>111</td> <td>17462</td> <td>26143</td> </tr> <tr> <td>xin_eng</td> <td>191</td> <td>834</td> <td>2518</td> <td>360714</td> <td>1744025</td> </tr> <tr> <td>TOTAL</td> <td>1010</td> <td>9325</td> <td>26348</td> <td>4032686</td> <td>9876086</td> </tr> </tbody> </table> <h3>资助说明</h3> 本项目部分由美国国防高级研究计划局（Defense Advanced Research Projects Agency）GALE项目资助，资助编号为HR0011-06-1-0003。本文档内容不必然代表美国政府的立场或政策，亦不应被视为官方背书。 <h3>示例</h3> 如需查看本语料库的示例数据，请参阅<a href="desc/addenda/LDC2011T07.jpg">此示例文件</a>。 本语料库部分内容 © 1994-2010 Agence France Presse, © 1994-2010 The Associated Press, © 1997-2010 Central News Agency (Taiwan), © 1994-1998, 2003-2009 Los Angeles Times-Washington Post News Service, Inc., © 1994-2010 New York Times, © 2010 The Washington Post News Service with Bloomberg News, © 1995-2010 Xinhua News Agency, © 2003, 2005, 2007, 2009, 2011 Trustees of the University of Pennsylvania

提供机构：

Linguistic Data Consortium

创建时间：

2020-11-30

搜集汇总

数据集介绍