biu-nlp/WEC-Eng
收藏资源简介：
# WEC-Eng A large-scale dataset for cross-document event coreference extracted from English Wikipedia. </br> - **Repository (Code for generating WEC):** https://github.com/AlonEirew/extract-wec - **Paper:** https://aclanthology.org/2021.naacl-main.198/ ### Languages English ## Load Dataset You can read in WEC-Eng files as follows (using the **huggingface_hub** library): ```json from huggingface_hub import hf_hub_url, cached_download import json REPO_ID = "datasets/biu-nlp/WEC-Eng" splits_files = ["Dev_Event_gold_mentions_validated.json", "Test_Event_gold_mentions_validated.json", "Train_Event_gold_mentions.json"] wec_eng = list() for split_file in splits_files: wec_eng.append(json.load(open(cached_download( hf_hub_url(REPO_ID, split_file)), "r"))) ``` ## Dataset Structure ### Data Splits - **Final version of the English CD event coreference dataset**<br> - Train - Train_Event_gold_mentions.json - Dev - Dev_Event_gold_mentions_validated.json - Test - Test_Event_gold_mentions_validated.json | | Train | Valid | Test | | ----- | ------ | ----- | ---- | | Clusters | 7,042 | 233 | 322 | | Event Mentions | 40,529 | 1250 | 1,893 | - **The non (within clusters) controlled version of the dataset (lexical diversity)**<br> - All (experimental) - All_Event_gold_mentions_unfiltered.json ### Data Instances ```json { "coref_chain": 2293469, "coref_link": "Family Values Tour 1998", "doc_id": "House of Pain", "mention_context": [ "From", "then", "on", ",", "the", "members", "continued", "their" ], "mention_head": "Tour", "mention_head_lemma": "Tour", "mention_head_pos": "PROPN", "mention_id": "108172", "mention_index": 1, "mention_ner": "UNK", "mention_type": 8, "predicted_coref_chain": null, "sent_id": 2, "tokens_number": [ 50, 51, 52, 53 ], "tokens_str": "Family Values Tour 1998", "topic_id": -1 } ``` ### Data Fields |Field|Value Type|Value| |---|:---:|---| |coref_chain|Numeric|Coreference chain/cluster ID| |coref_link|String|Coreference link wikipeida page/article title| |doc_id|String|Mention page/article title| |mention_context|List[String]|Tokenized mention paragraph (including mention)| |mention_head|String|Mention span head token| |mention_head_lemma|String|Mention span head token lemma| |mention_head_pos|String|Mention span head token POS| |mention_id|String|Mention id| |mention_index|Numeric|Mention index in json file| |mention_ner|String|Mention NER| |tokens_number|List[Numeric]|Mentions tokens ids within the context| |tokens_str|String|Mention span text| |topic_id|Ignore|Ignore| |mention_type|Ignore|Ignore| |predicted_coref_chain|Ignore|Ignore| |sent_id|Ignore|Ignore| ## Citation ``` @inproceedings{eirew-etal-2021-wec, title = "{WEC}: Deriving a Large-scale Cross-document Event Coreference dataset from {W}ikipedia", author = "Eirew, Alon and Cattan, Arie and Dagan, Ido", booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies", month = jun, year = "2021", address = "Online", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2021.naacl-main.198", doi = "10.18653/v1/2021.naacl-main.198", pages = "2498--2510", abstract = "Cross-document event coreference resolution is a foundational task for NLP applications involving multi-text processing. However, existing corpora for this task are scarce and relatively small, while annotating only modest-size clusters of documents belonging to the same topic. To complement these resources and enhance future research, we present Wikipedia Event Coreference (WEC), an efficient methodology for gathering a large-scale dataset for cross-document event coreference from Wikipedia, where coreference links are not restricted within predefined topics. We apply this methodology to the English Wikipedia and extract our large-scale WEC-Eng dataset. Notably, our dataset creation method is generic and can be applied with relatively little effort to other Wikipedia languages. To set baseline results, we develop an algorithm that adapts components of state-of-the-art models for within-document coreference resolution to the cross-document setting. Our model is suitably efficient and outperforms previously published state-of-the-art results for the task.", } ``` ## License We provide the following data sets under a <a href="https://creativecommons.org/licenses/by-sa/3.0/deed.en_US">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>. It is based on content extracted from Wikipedia that is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License ## Contact If you have any questions please create a Github issue at https://github.com/AlonEirew/extract-wec.
# WEC-Eng **WEC-Eng**：从英文维基百科（Wikipedia）中提取的大规模跨文档事件共指数据集。 - **代码仓库（WEC生成代码）**：https://github.com/AlonEirew/extract-wec - **相关论文**：https://aclanthology.org/2021.naacl-main.198/ ### 语言 英语 ## 数据集加载 你可以通过以下方式读取WEC-Eng文件（使用**huggingface_hub**库）： json from huggingface_hub import hf_hub_url, cached_download import json REPO_ID = "datasets/biu-nlp/WEC-Eng" splits_files = ["Dev_Event_gold_mentions_validated.json", "Test_Event_gold_mentions_validated.json", "Train_Event_gold_mentions.json"] wec_eng = list() for split_file in splits_files: wec_eng.append(json.load(open(cached_download( hf_hub_url(REPO_ID, split_file)), "r"))) ## 数据集结构 ### 数据划分 - **英文跨文档事件共指数据集最终版本**<br> - 训练集：Train_Event_gold_mentions.json - 开发集：Dev_Event_gold_mentions_validated.json - 测试集：Test_Event_gold_mentions_validated.json | | 训练集 | 开发集 | 测试集 | | ----- | ------ | ----- | ---- | | 簇数量 | 7,042 | 233 | 322 | | 事件提及数 | 40,529 | 1250 | 1,893 | - **数据集非（簇内）受控版本（词汇多样性实验）**<br> - 全量（实验用）：All_Event_gold_mentions_unfiltered.json ### 数据实例 以下为一条典型数据实例： json { "coref_chain": 2293469, "coref_link": "Family Values Tour 1998", "doc_id": "House of Pain", "mention_context": [ "From", "then", "on", ",", "the", "members", "continued", "their" ], "mention_head": "Tour", "mention_head_lemma": "Tour", "mention_head_pos": "PROPN", "mention_id": "108172", "mention_index": 1, "mention_ner": "UNK", "mention_type": 8, "predicted_coref_chain": null, "sent_id": 2, "tokens_number": [ 50, 51, 52, 53 ], "tokens_str": "Family Values Tour 1998", "topic_id": -1 } ### 数据字段 |字段名|值类型|字段说明| |---|:---:|---| |coref_chain|数值型|共指链/簇ID| |coref_link|字符串|共指链接对应的维基百科页面/文章标题| |doc_id|字符串|提及所在页面/文章标题| |mention_context|字符串列表|分词后的提及上下文（包含提及本身）| |mention_head|字符串|提及片段的中心词| |mention_head_lemma|字符串|提及片段中心词的词元| |mention_head_pos|字符串|提及片段中心词的词性标注| |mention_id|字符串|提及ID| |mention_index|数值型|提及在JSON文件中的索引| |mention_ner|字符串|提及的命名实体识别（NER）标签| |tokens_number|数值型列表|提及上下文内的令牌ID列表| |tokens_str|字符串|提及片段的文本内容| |topic_id|无意义|忽略字段| |mention_type|无意义|忽略字段| |predicted_coref_chain|无意义|忽略字段| |sent_id|无意义|忽略字段| ## 引用信息 @inproceedings{eirew-etal-2021-wec, title = "{WEC}：从维基百科提取大规模跨文档事件共指数据集", author = "Eirew, Alon and Cattan, Arie and Dagan, Ido", booktitle = "2021年国际计算语言学协会北美分会（NAACL）人类语言技术大会论文集", month = jun, year = "2021", address = "线上", publisher = "国际计算语言学协会（Association for Computational Linguistics）", url = "https://aclanthology.org/2021.naacl-main.198", doi = "10.18653/v1/2021.naacl-main.198", pages = "2498--2510", abstract = "跨文档事件共指消解是涉及多文本处理的自然语言处理（NLP）应用的基础任务。然而，现有该任务的语料库稀缺且规模相对较小，仅对属于同一主题的小规模文档簇进行标注。为补充这些资源并推动未来研究，我们提出维基百科事件共指数据集（WEC）：一种从维基百科提取大规模跨文档事件共指数据集的高效方法，其中共指链接不限于预定义主题范围内。我们将该方法应用于英文维基百科，提取得到大规模WEC-Eng数据集。值得注意的是，我们的数据集创建方法具有通用性，仅需较少工作量即可应用于其他语言的维基百科。为设置基准结果，我们开发了一种算法，将当前最先进的文档内共指消解模型组件适配至跨文档场景。我们的模型效率优异，且优于该任务此前已发表的最优结果。", } ## 授权协议 我们通过<a href="https://creativecommons.org/licenses/by-sa/3.0/deed.en_US">知识共享署名-相同方式共享3.0未移植版许可协议（Creative Commons Attribution-ShareAlike 3.0 Unported License）</a>发布以下数据集。本数据集基于从维基百科提取的内容，而维基百科内容采用知识共享署名-相同方式共享3.0未移植版许可协议进行授权。 ## 联系方式 如有任何疑问，请在https://github.com/AlonEirew/extract-wec提交GitHub Issue。
数据集概述
数据集名称
WEC-Eng
数据集描述
WEC-Eng是一个大规模的跨文档事件共指数据集，从英文维基百科中提取。
数据集语言
英语
数据集加载
使用huggingface_hub库加载数据集的示例代码如下：
json from huggingface_hub import hf_hub_url, cached_download import json REPO_ID = "datasets/biu-nlp/WEC-Eng" splits_files = ["Dev_Event_gold_mentions_validated.json", "Test_Event_gold_mentions_validated.json", "Train_Event_gold_mentions.json"] wec_eng = list() for split_file in splits_files: wec_eng.append(json.load(open(cached_download( hf_hub_url(REPO_ID, split_file)), "r")))
数据集结构
数据分割
- 最终版本的英文CD事件共指数据集
- 训练集 - Train_Event_gold_mentions.json
- 开发集 - Dev_Event_gold_mentions_validated.json
- 测试集 - Test_Event_gold_mentions_validated.json
| 训练 | 验证 | 测试 | |
|---|---|---|---|
| 集群 | 7,042 | 233 | 322 |
| 事件提及 | 40,529 | 1,250 | 1,893 |
- 数据集的非控制版本（词汇多样性）
- 全部（实验性） - All_Event_gold_mentions_unfiltered.json
数据实例
json { "coref_chain": 2293469, "coref_link": "Family Values Tour 1998", "doc_id": "House of Pain", "mention_context": [ "From", "then", "on", ",", "the", "members", "continued", "their" ], "mention_head": "Tour", "mention_head_lemma": "Tour", "mention_head_pos": "PROPN", "mention_id": "108172", "mention_index": 1, "mention_ner": "UNK", "mention_type": 8, "predicted_coref_chain": null, "sent_id": 2, "tokens_number": [ 50, 51, 52, 53 ], "tokens_str": "Family Values Tour 1998", "topic_id": -1 }
数据字段
| 字段 | 值类型 | 值描述 |
|---|---|---|
| coref_chain | Numeric | 共指链/集群ID |
| coref_link | String | 共指链接维基百科页面/文章标题 |
| doc_id | String | 提及页面/文章标题 |
| mention_context | List[String] | 提及段落的标记化文本（包括提及） |
| mention_head | String | 提及跨度头部标记 |
| mention_head_lemma | String | 提及跨度头部标记词元 |
| mention_head_pos | String | 提及跨度头部标记POS |
| mention_id | String | 提及ID |
| mention_index | Numeric | 提及在JSON文件中的索引 |
| mention_ner | String | 提及NER |
| tokens_number | List[Numeric] | 提及上下文中的标记ID |
| tokens_str | String | 提及跨度文本 |
| topic_id | Ignore | 忽略 |
| mention_type | Ignore | 忽略 |
| predicted_coref_chain | Ignore | 忽略 |
| sent_id | Ignore | 忽略 |
引用信息
@inproceedings{eirew-etal-2021-wec, title = "{WEC}: Deriving a Large-scale Cross-document Event Coreference dataset from {W}ikipedia", author = "Eirew, Alon and Cattan, Arie and Dagan, Ido", booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies", month = jun, year = "2021", address = "Online", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2021.naacl-main.198", doi = "10.18653/v1/2021.naacl-main.198", pages = "2498--2510", abstract = "Cross-document event coreference resolution is a foundational task for NLP applications involving multi-text processing. However, existing corpora for this task are scarce and relatively small, while annotating only modest-size clusters of documents belonging to the same topic. To complement these resources and enhance future research, we present Wikipedia Event Coreference (WEC), an efficient methodology for gathering a large-scale dataset for cross-document event coreference from Wikipedia, where coreference links are not restricted within predefined topics. We apply this methodology to the English Wikipedia and extract our large-scale WEC-Eng dataset. Notably, our dataset creation method is generic and can be applied with relatively little effort to other Wikipedia languages. To set baseline results, we develop an algorithm that adapts components of state-of-the-art models for within-document coreference resolution to the cross-document setting. Our model is suitably efficient and outperforms previously published state-of-the-art results for the task.", }
许可证
本数据集根据<a href="https://creativecommons.org/licenses/by-sa/3.0/deed.en_US">Creative Commons Attribution-ShareAlike 3.0 Unported License</a>提供。数据基于从维基百科提取的内容，该内容根据Creative Commons Attribution-ShareAlike 3.0 Unported License授权。
