TDT2 Multilanguage Text Version 4.0

<h3>Introduction</h3><br>
<p>Topic Detection and Tracking (TDT) refers to automatic techniques for finding topically related material in streams of data such as newswire and broadcast news. The TDT2 corpus was created to support three TDT2 tasks: find topically homogeneous sections (segmentation), detect the occurrrence of new events (detection), and track the reoccurrencce of old or new events (tracking).&nbsp;</p><br>
<h3>Data</h3><br>
<p>TDT2 Multilanguage Text Corpus Version 4.0 contains news data collected daily from nine news sources in two languages (American English and Mandarin Chinese), over a period of six months (January - June 1998). Both manually-created reference text and automatically- generated text (ASR and/or machine translation) are provided for all broadcast and all Mandarin data.</p><br>
<p>This version has been prepared to complement the first general release of the TDT3 Multilanguage Text Corpus, providing new enhancements to make the data content more accessible to a broader research community. The news sources and approximate number of stories per source (in thousands) are as follows:</p><br>
<p>English sources (thousands of stories)</p><br>
<p>New York Times Newswire Service 11.8</p><br>
<p>Associated Press Worldstream Service 12.8</p><br>
<p>Cable News Network, Headline News 15.8</p><br>
<p>American Broadcasting Co., World News Tonight 2.1</p><br>
<p>Public Radio International, The World 2.9</p><br>
<p>Voice of America (news programs) 8.2</p><br>
<p>Total English stories: 53.6 thousand)</p><br>
<p>Mandarin sources (thousands of stories)</p><br>
<p>Xinhua News Agency 11.3</p><br>
<p>Zaobao News Agency 5.2</p><br>
<p>Voice of America (news programs) 2.3</p><br>
<p>Total Mandarin stories: 18.8 thousand</p><br>
<p>This release consists of the English and Mandarin text components of the TDT2 corpus. The data was collected daily over a period of six months (January-June 1998) from the following sources.</p><br>
<ul><br>
<li>American Broadcasting Company (ABC)</li><br>
<li>Associated Press</li><br>
<li>Cable News Network, Inc. (CNN)</li><br>
<li>New York Times</li><br>
<li>Public Radio International (PRI)</li><br>
<li>Voice of America (VOA)</li><br>
<li>Xinhua News Agency</li><br>
<li>ZaoBao News</li><br>
</ul><br>
<p>The data is provided in the following formats.</p><br>
<p>.sgm: Reference true-text, with markup providing story boundaries and descriptive information .tkn: Tokenized version of sgml data, with all descriptive and boundary information removed .as0: Output of the Dragon ASR system in tokenized form with information on timing, speaker clusters, and confidence .as1: Output of the BBN ASR system in tokenized form with timing information (English Only) .mttkn: SYSTRAN output from .tkn (Mandarin Only) .mtas0: SYSTRAN output from .as0 (Mandarin Only)</p><br>
<p>The corpus also includes topic relevance tables as well as tables for locating story boundaries.</p><br>
<h3>Updates</h3><br>
<p>7/21/16 - Topic tables were added to the release and the online documentation folder.</p><br>
<p>&nbsp;</p><br>
<p>The World is a co-production of Public Radio International and the British Broadcasting Corporation and is produced at WGBH Boston.</p></br>
Portions © 1998 American Broadcasting Company, The Associated Press, Cable News Network, LP, LLLP, New York Times, Public Radio International, SPH AsiaOne Ltd, Xinhua News Agency, © 1998-2001 Trustees of the University of Pennsylvania <br><br>The World is a co-production of Public Radio International and the British Broadcasting Corporation and is produced at WGBH Boston.