TDT3 Multilanguage Text Version 2.0

Introduction
Topic Detection and Tracking (TDT) refers to automatic techniques for finding topically related material in streams of data such as newswire and broadcast news. The TDT3 corpus was created to support three TDT3 tasks: to find topically homogeneous sections (segmentation), to detect the occurrence of new events (detection) and to track the reoccurrence of old or new events (tracking).

Data
TDT3 Multilanguage Text Corpus Version 2.0 is the first general release of this collection (Version 1.0 was made available only to participants in the TDT 1999 and 2000 evaluation tests). It contains data from the same nine sources found in TDT2, plus two additional English television sources. Like TDT2, it provides both manually-created and automatically-generated text for most sources.

For TDT3, the daily collection took place over a period of three months (October - December 1998). The sources and approximate number of stories per source are as follows:

English sources Thousands of stories

New York Times Newswire Service 6.9

Associated Press Worldstream Service 7.3

Cable News Network, "Headline News" 9.0

American Broadcasting Co., "World News Tonight" 1.0

Public Radio International, "The World" 1.6

Voice of America, English news programs 3.9

MS-NBC, "News with Brian Williams" 0.7

National Broadcasting Co., "NBC Nightly News" 0.8

Total English stories: 31.2 thousand

Mandarin sources Thousands of stories

Xinhua News Agency 5.2

Zaobao News Agency 3.8

Voice of America, Mandarin Chinese news programs 3.8

Total Mandarin stories: 12.8 thousand

The goal of Topic Detection and Tracking - Phase 3 (TDT3) is to create core technology to monitor multiple streams of news in multiple languages and media (newswire, radio, television, web sites or some future combination or innovation), segmenting the streams into individual stories, detecting new topics and tracking all stories discussing them. In additional to the TDT2 tasks of segmentation, detection and tracking, TDT3 adds the tasks of first story detection and story-link detection. The goal of the latter is to detect links between stories that discuss the same topic even though the topic has not been defined in advance.

There are two types of files in this publication:

asr_sgm -- text data output from automatic speech recognition (ASR) systems in English and Mandarin, formatted in "TIPSTER- style" SGML, derived from the audio recordings of radio and TV broadcasts.

tkn_sgm -- reference text data (newswire, closed captions and manual transcripts), formatted in "TIPSTER-style" SGML

Samples
Please view this asr_sgm sample and tkn_sgm sample.

Updates
07/21/16 - Topic tables added.

07/01/16 - Topic tables updated to v4.0.

Copyright
Portions © 1998 American Broadcasting Company, The Associated Press, Cable News Network, LP, LLLP, National Broadcasting Company, Inc., New York Times, Public Radio International, SPH AsiaOne Ltd, Xinhua News Agency, © 1998-2001 Trustees of the University of Pennsylvania

The World is a co-production of Public Radio International and the British Broadcasting Corporation and is produced at WGBH Boston.

引言
话题检测与跟踪（Topic Detection and Tracking，TDT）指在新闻专线、广播电视等数据流中挖掘主题相关内容的自动化技术。TDT3语料库旨在支撑三类TDT3任务：划分主题同质片段（分段任务）、检测新事件的出现（检测任务），以及跟踪新旧事件的重现（跟踪任务）。

数据集详情
TDT3多语言文本语料库2.0版是该集合的首次公开发布（1.0版仅对1999年与2000年TDT评估测试的参与者开放）。其数据源涵盖TDT2的全部9类来源，外加两类新增英语电视数据源。与TDT2一致，绝大多数数据源均同时提供人工生成文本与自动生成文本。

TDT3的每日数据采集周期为1998年10月至12月，共计三个月。各数据源及单源报道的大致数量如下：

### 英语数据源（单位：千篇报道）
纽约时报新闻专线服务：6.9
美联社全球专线服务：7.3
有线电视新闻网"头条新闻"：9.0
美国广播公司"世界新闻今晚"：1.0
国际公共广播"全球新闻"：1.6
美国之音英语新闻节目：3.9
微软全国广播公司"布莱恩·威廉姆斯新闻"：0.7
全国广播公司"NBC晚间新闻"：0.8
英语报道总计：31.2千篇

### 普通话数据源（单位：千篇报道）
新华社：5.2
联合早报通讯社：3.8
美国之音普通话新闻节目：3.8
普通话报道总计：12.8千篇

话题检测与跟踪第三阶段（TDT3）的目标是研发核心技术，以监测多语言、多媒介的新闻数据流（包括新闻专线、广播、电视、网站或未来的其他组合与创新形式），将数据流划分为独立单篇报道，检测新话题并跟踪所有讨论该话题的报道。除TDT2的分段、检测与跟踪任务外，TDT3新增首次报道检测与报道关联检测任务。后者的目标是识别讨论同一话题的报道之间的关联，即便该话题未预先定义。

本次发布包含两类文件：
asr_sgm：源自广播电视音频录制的自动语音识别（Automatic Speech Recognition，ASR）系统输出的英语与普通话文本数据，采用"TIPSTER风格"SGML格式。
tkn_sgm：参考文本数据（涵盖新闻专线、隐藏式字幕与人工转录文本），采用"TIPSTER风格"SGML格式。

示例
请查看此asr_sgm示例与tkn_sgm示例。

更新记录
2016年7月21日：新增话题表格。
2016年7月1日：话题表格更新至v4.0版本。

版权声明
部分内容©1998 美国广播公司、美联社、有线电视新闻网有限合伙公司、全国广播公司、纽约时报、国际公共广播、新加坡报业控股AsiaOne有限公司、新华社；©1998-2001 宾夕法尼亚大学董事会。
《全球新闻》是国际公共广播与英国广播公司的联合制作项目，由波士顿WGBH电台出品。

搜集汇总

背景与挑战

背景概述

TDT3 Multilanguage Text Version 2.0是一个多语言新闻文本语料库，包含英语和普通话的自动语音识别输出及参考文本，用于支持主题检测和跟踪技术的研究。数据集覆盖了1998年10月至12月的新闻内容，来自多个新闻源，包括纽约时报、美联社、CNN等。