LORELEI Uyghur Incident Language Pack

<h3><strong>Introduction</strong></h3>
<p>LORELEI Uyghur Incident Language Pack (LDC2024T07) was developed by the Linguistic Data Consortium and consists of approximately 28 million words of Uyghur monolingual text, 500,000 words of English monolingual text, 3.3 million words of parallel and comparable Uyghur-English text, and 200,000 words of data annotated for Simple Named Entities and Situation Frames. It contains all of the text data, annotations, supplemental resources and related software tools for the Uyghur language that were used in the <a href="https://www.nist.gov/itl/iad/mig/lorehlt-evaluations">DARPA LORELEI / LoReHLT 2016 Evaluation</a>.</p>
<p>The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations like natural disasters or disease outbreaks. Linguistic resources for LORELEI include Representative Language Packs and Incident Language Packs for over two dozen low resource languages, comprising data, annotations, basic natural language processing tools, lexicons and grammatical resources. Representative languages were selected to provide broad typological coverage, while incident languages were selected to evaluate system performance on a language whose identity was disclosed at the start of the evaluation.</p>
<p>The evaluation protocol was based on a scenario in which an unforeseen event triggered a need for humanitarian and logistical support in a region where the incident language had received little or no attention in natural language processing (NLP) research. Evaluation participants provided NLP solutions, including information extraction and machine translation, based on limited resources and with very little time for development.</p>
<h3><strong>Data</strong></h3>
<p>Uyghur is spoken mainly in northwestern China, as well as in Kazakhstan, Kyrgyzstan, and Uzbekistan. Data was collected in the following genres: news, social network, weblog, newsgroup, discussion forum, and reference material.</p>
<p>Named entity annotation identified entities to be detected by systems for scoring purposes. Situation frame analysis was designed to extract basic information about needs and relevant issues for planning a disaster response effort.</p>
<p>Also included in this release are lexical and grammatical resources as well as three tools: two to recreate original source data from the processed XML material and the other to condition text data users download from Twitter.</p>
<p>Monolingual, parallel and comparable text are presented in XML with associated dtds. Situation frame annotation data is presented as tab delimited files. All text is UTF-8 encoded.</p>
<p>The knowledge base for entity linking annotation for this corpus and all LORELEI Representative Language and Incident Language Packs is available separately as&nbsp;<a href="../../../LDC2020T10">LORELEI Entity Detection and Linking Knowledge Base (LDC2020T10)</a>.</p>
<h3>Sponsorship</h3>
<p>This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-15-C-0123. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of DARPA.</p>
<h3>Samples</h3>
<p>Please view these samples:</p>
<ul>
<li><a href="desc/addenda/LDC2024T07.eng.ltf.xml">English LTF XML</a></li>
<li><a href="desc/addenda/LDC2024T07.eng.psm.xml">English PSM XML</a></li>
<li><a href="desc/addenda/LDC2024T07.il3.ltf.xml">Uyghur LTF XML</a></li>
<li><a href="desc/addenda/LDC2024T07.il3.psm.xml">Uyghur PSM XML</a></li>
<li><a href="desc/addenda/LDC2024T07.laf">Full Entity Annotation XML</a></li>
<li><a href="desc/addenda/LDC2024T07.entity.tab">Mentions Annotation XML</a></li>
<li><a href="desc/addenda/LDC2024T07.needs.tab">Needs Annotation XML</a></li>
</ul>
<h3>Updates</h3>
<p>None at this time.</p>

<h3><strong>引言</strong></h3><p>应急事件低资源语言（Low Resource Languages for Emergent Incidents, LORELEI）维吾尔语专项语言包（LDC2024T07）由语言数据联盟（Linguistic Data Consortium, LDC）开发，包含约2800万词维吾尔语单语文本、50万词英语单语文本、330万词维英平行及可比语料，以及20万词标注有简单命名实体与场景框架的语料。本语言包涵盖了<a href="https://www.nist.gov/itl/iad/mig/lorehlt-evaluations">DARPA LORELEI / LoReHLT 2016评估</a>中所用的全部维吾尔语文本数据、标注集、辅助资源及相关软件工具。</p><p>应急事件低资源语言（Low Resource Languages for Emergent Incidents, LORELEI）项目旨在针对自然灾害、疾病暴发等突发场景下的低资源语言开发人类语言技术。该项目的语言资源涵盖二十余种低资源语言的代表性语言包与专项语言包，包含语料、标注集、基础自然语言处理（Natural Language Processing, NLP）工具、词典及语法资源。其中代表性语言的遴选以覆盖广泛类型学特征为目标，而专项语言则用于评估系统在评估启动时才公布身份的语言上的性能表现。</p><p>本次评估的协议基于如下场景：某一未预见事件触发了某一地区的人道主义与后勤支援需求，而该地区所用语言在自然语言处理（NLP）研究中几乎未受到关注。评估参与者需基于有限资源与极短开发周期，提供包括信息抽取与机器翻译在内的自然语言处理解决方案。</p><h3><strong>语料概况</strong></h3><p>维吾尔语主要使用于中国西北地区，以及哈萨克斯坦、吉尔吉斯斯坦与乌兹别克斯坦。本次采集的语料涵盖以下体裁：新闻、社交网络、博客、新闻组、讨论论坛及参考资料。</p><p>命名实体标注用于识别系统需检测并参与评分的实体。场景框架分析旨在提取灾害应对规划所需的需求与相关议题的基础信息。</p><p>本发布包还包含词汇与语法资源，以及三款工具：两款用于从处理后的XML源数据重建原始语料，另一款用于处理用户从Twitter下载的文本数据。</p><p>单语、平行及可比语料以XML格式存储，并附带对应的文档类型定义（Document Type Definition, DTD）。场景框架标注数据以制表符分隔文件形式提供。所有文本均采用UTF-8编码。</p><p>本语料库及所有LORELEI代表性语言包与专项语言包的实体链接标注知识库，可作为<a href="../../../LDC2020T10">LORELEI实体检测与链接知识库（LDC2020T10）</a>单独获取。</p><h3>资助说明</h3><p>本材料基于美国国防高级研究计划局（Defense Advanced Research Projects Agency, DARPA）合同HR0011-15-C-0123资助的研究工作。本材料中表达的任何观点、发现、结论或建议均为作者本人的观点，不一定反映DARPA的官方立场。</p><h3>示例文件</h3><p>请查看以下示例：</p><ul><li><a href="desc/addenda/LDC2024T07.eng.ltf.xml">英语LTF XML</a></li><li><a href="desc/addenda/LDC2024T07.eng.psm.xml">英语PSM XML</a></li><li><a href="desc/addenda/LDC2024T07.il3.ltf.xml">维吾尔语LTF XML</a></li><li><a href="desc/addenda/LDC2024T07.il3.psm.xml">维吾尔语PSM XML</a></li><li><a href="desc/addenda/LDC2024T07.laf">完整实体标注XML</a></li><li><a href="desc/addenda/LDC2024T07.entity.tab">提及标注XML</a></li><li><a href="desc/addenda/LDC2024T07.needs.tab">需求标注XML</a></li></ul><h3>更新记录</h3><p>暂无更新。</p>

搜集汇总

背景与挑战

背景概述

该数据集是LORELEI项目下的维吾尔语紧急事件语言包，包含约2800万单词维吾尔语文本、50万单词英语文本，以及330万单词维英平行和可比文本，附带命名实体和情境框架标注。数据来源于新闻、社交媒体等多种类型，旨在支持低资源语言在自然灾害等突发事件中的自然语言处理技术评估和开发。