coref-data/korean_ecmt_raw

--- license: cc-by-nc-sa-4.0 --- # Korean Effective Crowdsourcing of Multiple Tasks (ECMT) for Comprehensive Knowledge Extraction - Project: https://github.com/machinereading/crowdsourcing - Data source: https://figshare.com/s/7367aeca244efae03068 ## Details Annotated text from Korean Wikipedia and KBox (Korean DBpedia). Includes a crowd sourced training set and expert annotated (reviewed by four experts) test set. The dataset was annotated by crowdworks in multiple stages. * Phase I: entity mention detection annotation; candidate entity mentions are selected in a text * Phase II: entity linking annotation; candidate mentions can be linked to a knowledge base * Phase III: coreference annotation; entities can be linked to pronouns, demonstrative determiners, and antecedent mentions * Phase IV: relation extraction annotation; relations between entities are annotated ### Annotation Notes #### Phase I * For each mention, the annotator selects a category from one of 16 options: person, study field, theory, artifact, organization, location, civilization, event, year, time, quantity, job, animal, plant, material, and term. * Entities can be things, concepts, ideas, or events: ``` 개체란 다른 것들과 분리되어 존재하는 것으로, 개체는 물질적 존재일 필요는 없으며 개념적 아이디어 혹은 사건도 될 수 있다 개체의 대표적인 범주에는 사람, 물체, 조직, 기관, 장소, 시간, 사건 등이 포함된다 ``` * Compound nouns are tagged with the largest span: ``` 복합명사인 경우 가장 넓은 단위로 태깅해주세요 ex) [상하이] [디즈니랜드] -> [상하이 디즈니랜드] ``` * Final result is created by merging annotations from two separate annotators. #### Phase II * For each mention, a list of candidates from the knowledge base are shown. The annotator can select a candidate, not in candidate list, or not an entity. * Each document was annotated by a single annotator. #### Phase III * For each mention, the annotator can select a preceding mention, no antecedent, or error. Noun phrases and pronouns are extracted using the parse information. * "We scaled down the coreference resolution by limiting the scope of the target mentions to a named entity, pronoun, and definite noun phrase." * Postfixes particles (조사) are not included in the antecedent: ``` [작업대상] 아래 항목에서 조사등을 제외(교정)해 주세요. 그녀는 -> 그녀 ``` ## Citation ``` @inproceedings{nam-etal-2020-effective, title = "Effective Crowdsourcing of Multiple Tasks for Comprehensive Knowledge Extraction", author = "Nam, Sangha and Lee, Minho and Kim, Donghwan and Han, Kijong and Kim, Kuntae and Yoon, Sooji and Kim, Eun-kyung and Choi, Key-Sun", editor = "Calzolari, Nicoletta and B{\'e}chet, Fr{\'e}d{\'e}ric and Blache, Philippe and Choukri, Khalid and Cieri, Christopher and Declerck, Thierry and Goggi, Sara and Isahara, Hitoshi and Maegaard, Bente and Mariani, Joseph and Mazo, H{\'e}l{\`e}ne and Moreno, Asuncion and Odijk, Jan and Piperidis, Stelios", booktitle = "Proceedings of the Twelfth Language Resources and Evaluation Conference", month = may, year = "2020", address = "Marseille, France", publisher = "European Language Resources Association", url = "https://aclanthology.org/2020.lrec-1.27", pages = "212--219", abstract = "Information extraction from unstructured texts plays a vital role in the field of natural language processing. Although there has been extensive research into each information extraction task (i.e., entity linking, coreference resolution, and relation extraction), data are not available for a continuous and coherent evaluation of all information extraction tasks in a comprehensive framework. Given that each task is performed and evaluated with a different dataset, analyzing the effect of the previous task on the next task with a single dataset throughout the information extraction process is impossible. This paper aims to propose a Korean information extraction initiative point and promote research in this field by presenting crowdsourcing data collected for four information extraction tasks from the same corpus and the training and evaluation results for each task of a state-of-the-art model. These machine learning data for Korean information extraction are the first of their kind, and there are plans to continuously increase the data volume. The test results will serve as an initiative result for each Korean information extraction task and are expected to serve as a comparison target for various studies on Korean information extraction using the data collected in this study.", language = "English", ISBN = "979-10-95546-34-4", } ```--- 许可证：CC BY-NC-SA 4.0（知识共享署名-非商业性使用-相同方式共享4.0协议） --- # 面向综合知识抽取的多任务高效众包韩语数据集（ECMT，Korean Effective Crowdsourcing of Multiple Tasks） - 项目地址：https://github.com/machinereading/crowdsourcing - 数据源地址：https://figshare.com/s/7367aeca244efae03068 ## 数据集详情 本数据集的标注文本源自韩语维基百科与KBox（韩语DBpedia），包含众包构建的训练集，以及经四位专家审核标注的测试集。该数据集由CrowdWorks分多阶段完成标注： * 第一阶段：实体提及检测标注（entity mention detection annotation）：从文本中筛选候选实体提及 * 第二阶段：实体链接标注（entity linking annotation）：将候选提及关联至知识库 * 第三阶段：共指消解标注（coreference annotation）：建立实体与代词、指示限定词及先行提及的关联 * 第四阶段：关系抽取标注（relation extraction annotation）：标注实体间的语义关系 ### 标注说明 #### 第一阶段 * 针对每个实体提及，标注者需从16个预设类别中选择其一：人物、研究领域、理论、人工制品、组织机构、地点、文明、事件、年份、时间、数量、职业、动物、植物、材料、术语。 * 实体可涵盖事物、概念、思想或事件： 所谓实体，是指与其他事物相区别而独立存在的对象，实体不必具有物质性，也可以是概念性思想或事件。实体的典型范畴包括人物、物体、组织、机构、地点、时间、事件等。 * 复合名词需以最大语义跨度进行标注，示例规则： 复合名词标注请以最大语义单元进行标记，例如：[上海] [迪士尼乐园] → [上海迪士尼乐园] * 最终标注结果由两位独立标注者的标注结果合并得到。 #### 第二阶段 * 针对每个实体提及，系统将展示知识库中的候选实体列表，标注者可选择列表内的候选条目、手动输入非列表内的候选条目，或判定该提及并非实体。 * 每份文档仅由一位标注者完成标注。 #### 第三阶段 * 针对每个实体提及，标注者可选择其先行提及、判定无先行项，或标记为标注错误。通过句法解析信息提取名词短语与代词。 * "我们将共指消解的任务范围限定在命名实体、代词与定指名词短语内，以此简化任务。" * 朝鲜语后置助词（조사）不应被纳入先行提及范围： 【标注要求】请从标注对象中剔除助词类成分并进行校正，示例：그녀는 → 그녀（即「她是」→「她」） ## 引用 @inproceedings{nam-etal-2020-effective, title = "面向综合知识抽取的多任务高效众包", author = "Nam, Sangha and Lee, Minho and Kim, Donghwan and Han, Kijong and Kim, Kuntae and Yoon, Sooji and Kim, Eun-kyung and Choi, Key-Sun", editor = "Calzolari, Nicoletta and Béchet, Frédéric and Blache, Philippe and Choukri, Khalid and Cieri, Christopher and Declerck, Thierry and Goggi, Sara and Isahara, Hitoshi and Maegaard, Bente and Mariani, Joseph and Mazo, Hélène and Moreno, Asuncion and Odijk, Jan and Piperidis, Stelios", booktitle = "第十二届语言资源与评估会议论文集", month = may, year = "2020", address = "Marseille, France", publisher = "European Language Resources Association", url = "https://aclanthology.org/2020.lrec-1.27", pages = "212--219", abstract = "从非结构化文本中抽取信息在自然语言处理领域发挥着至关重要的作用。尽管现有研究已针对各项信息抽取任务（即实体链接、共指消解与关系抽取）展开了广泛探索，但目前尚无数据集可支撑在统一框架下对所有信息抽取任务进行连贯且持续的评估。由于每项任务均使用独立数据集完成与评估，因此无法依托单一数据集贯穿整个信息抽取流程，分析前序任务对后续任务的影响。本文通过发布源自同一语料库、适用于四项信息抽取任务的众包数据，以及当前最优模型在各项任务上的训练与评估结果，旨在提出韩语信息抽取领域的基准方案并推动该领域研究。这批面向韩语信息抽取的机器学习数据集尚属同类首例，团队计划持续扩充数据集规模。本次测试结果将作为各项韩语信息抽取任务的基准结果，同时有望成为后续各类基于本数据集开展的韩语信息抽取研究的对比基准。", language = "English", ISBN = "979-10-95546-34-4", }