LORELEI Tamil Representative Language Pack

<h3>Introduction</h3>
<p>LORELEI Tamil Representative Language Pack (LDC2023T03) consists of Tamil monolingual text, Tamil-English parallel text, annotations, supplemental resources and related software tools developed by the Linguistic Data Consortium (LDC) for the DARPA LORELEI program.</p>
<p>The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations like natural disasters or disease outbreaks. Linguistic resources for LORELEI include Representative Language Packs and Incident Language Packs for over two dozen low resource languages, comprising data, annotations, basic natural language processing tools, lexicons and grammatical resources. Representative languages were selected to provide broad typological coverage, while incident languages were selected to evaluate system performance on a language whose identity was disclosed at the start of the evaluation.</p>
<h3>Data</h3>
<p>Tamil is spoken mainly in South Asia. It is an official language of the Indian state of Tamil Nadu, Sri Lanka, Singapore, and the Union Territory of Puducherry. Data was collected in the following genres: discussion forum, news, reference, social network, and weblogs. Both monolingual text collection and parallel text creation involved a combination of manual and automatic methods.</p>
<p>Data volumes are as follows:</p>
<ul>
<li>Over 41 million words of Tamil monolingual text, over 226,000 of which were translated into English</li>
<li>680,000 words of found Tamil-English parallel text</li>
<li>81,000 Tamil words translated from English data</li>
</ul>
<p>Over 78,000 words are annotated for named entities, and over 24,000 words were annotated for full entity including nominals and pronouns, simple semantic annotation, situation frame annotation, and entity linking.</p>
<p>Lexical resources and software tools are also included in this release. The tools recreate original source data from the processed XML material, condition text data users download from Twitter, apply sentence segmentation to raw text, and support named entity tagging.</p>
<p>Monolingual and parallel text are presented in XML with associated dtds. Annotation data is presented as tab delimited files or XML. All text is UTF-8 encoded.</p>
<p>The knowledge base for entity linking annotation for this corpus and all LORELEI Representative Language and Incident Language Packs is available separately as&nbsp;<a href="../../../LDC2020T10">LORELEI Entity Detection and Linking Knowledge Base (LDC2020T10)</a>.</p>
<h3>Sponsorship</h3>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-15-C-0123. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of DARPA.</p>
<h3>Samples</h3>
<p>Please view these samples:</p>
<ul>
<li><a href="desc/addenda/LDC2023T02.aln.xml">Translation Alignment XML</a></li>
<li><a href="desc/addenda/LDC2023T03.eng.ltf.xml">English LTF XML</a></li>
<li><a href="desc/addenda/LDC2023T03.eng.psm.xml">English PSM XML</a></li>
<li><a href="desc/addenda/LDC2023T03.ltf.tam.xml">Tamil LTF XML</a></li>
<li><a href="desc/addenda/LDC2023T03.psm.tam.xml">Tamil PSM XML</a></li>
<li><a href="desc/addenda/LDC2023T03.issues.tab">Issues Table (TXT)</a></li>
<li><a href="desc/addenda/LDC2023T03.entity.tab">Entity Table (TXT)</a></li>
<li><a href="desc/addenda/LDC2023T03.full.laf.xml">Full Entity Annotation XML</a></li>
<li><a href="desc/addenda/LDC2023T03.sem.laf.xml">Semantic Annotation XML</a></li>
</ul>
<h3>Updates</h3>
<p>None at this time.</p>

<h3>引言</h3>
<p>LORELEI泰米尔语代表语言包（LDC2023T03）由语言数据联盟（Linguistic Data Consortium, LDC）为美国国防高级研究计划局（Defense Advanced Research Projects Agency, DARPA）的LORELEI项目开发，包含泰米尔语单语文本、泰米尔语-英语平行文本、标注数据、补充资源及相关软件工具。</p>
<p>LORELEI项目全称为Low Resource Languages for Emergent Incidents（低资源语言应对突发事件），旨在针对自然灾害、疾病暴发等突发场景，为低资源语言开发人类语言技术。该项目的语言资源涵盖二十余种低资源语言的代表语言包与事件语言包，包含数据、标注、基础自然语言处理工具、词典及语法资源。其中代表语言的选取以覆盖广泛的类型学特征为目标，而事件语言则用于评估系统在评估启动时才公布具体语种的场景下的性能表现。</p>
<h3>数据</h3>
<p>泰米尔语主要通行于南亚地区，是印度泰米尔纳德邦、斯里兰卡、新加坡以及本地治里联邦属地的官方语言。本次数据集采集了以下体裁的文本：论坛讨论、新闻、参考资料、社交网络内容及博客。单语文本采集与平行文本创建均结合了人工与自动两种处理方法。</p>
<p>数据规模如下：</p>
<ul>
<li>超过4100万词的泰米尔语单语文本，其中超22.6万词被译为英语</li>
<li>68万词的现有泰米尔语-英语平行文本</li>
<li>8.1万词的泰米尔语译文（源自英语语料）</li>
</ul>
<p>超7.8万词的文本完成了命名实体标注，超2.4万词的文本完成了全实体标注，涵盖名词性成分与代词、完整语义标注、场景框架标注以及实体链接标注。</p>
<p>本次发布包还包含词汇资源与软件工具。这些工具可从处理后的XML材料中还原原始源数据，对用户从Twitter下载的文本数据进行预处理，对原始文本执行分句操作，并支持命名实体标记任务。</p>
<p>单语与平行文本以XML格式呈现，并附带对应的文档类型定义（Document Type Definition, DTD）。标注数据以制表符分隔文件或XML格式提供。所有文本均采用UTF-8编码。</p>
<p>本语料库以及所有LORELEI代表语言包、事件语言包的实体链接标注知识库，可通过<a href="../../../LDC2020T10">LORELEI实体检测与链接知识库（LDC2020T10）</a>单独获取。</p>
<h3>资助说明</h3>
<p>本材料基于美国国防高级研究计划局（DARPA）合同HR0011-15-C-0123资助的研究工作成果。本材料中表达的任何观点、发现、结论或建议仅代表作者本人，不一定反映DARPA的官方立场。</p>
<h3>示例</h3>
<p>请查看以下示例文件：</p>
<ul>
<li><a href="desc/addenda/LDC2023T02.aln.xml">翻译对齐XML文件</a></li>
<li><a href="desc/addenda/LDC2023T03.eng.ltf.xml">英语LTF XML文件</a></li>
<li><a href="desc/addenda/LDC2023T03.eng.psm.xml">英语PSM XML文件</a></li>
<li><a href="desc/addenda/LDC2023T03.ltf.tam.xml">泰米尔语LTF XML文件</a></li>
<li><a href="desc/addenda/LDC2023T03.psm.tam.xml">泰米尔语PSM XML文件</a></li>
<li><a href="desc/addenda/LDC2023T03.issues.tab">问题表（TXT）</a></li>
<li><a href="desc/addenda/LDC2023T03.entity.tab">实体表（TXT）</a></li>
<li><a href="desc/addenda/LDC2023T03.full.laf.xml">全实体标注XML文件</a></li>
<li><a href="desc/addenda/LDC2023T03.sem.laf.xml">语义标注XML文件</a></li>
</ul>
<h3>更新记录</h3>
<p>暂无更新。</p>

搜集汇总

背景与挑战

背景概述

该数据集是LORELEI项目的一部分，专注于泰米尔语这一低资源语言，提供大规模单语和平行文本数据，涵盖讨论论坛、新闻和博客等多种来源。数据集包含超过4100万词的泰米尔语文本以及相关翻译，并附带实体标注、语义注释和软件工具，适用于机器翻译、信息提取等自然语言处理应用。