coref-data/niv2_winogrande_raw

--- license: apache-2.0 --- # Natural Instructions v2 Winogrande Tasks - Project: https://github.com/allenai/natural-instructions - Data source: [DataProvenanceInitiative/niv2_submix_original](https://huggingface.co/datasets/DataProvenanceInitiative/niv2_submix_original) ## Details This dataset contains all Winogrande examples that were included in the [Flan 2022 collection](https://github.com/google-research/FLAN/tree/main/flan/v2) which were orignally published in Super-Natural-Instructions. The data is copied from the preprocessed Natural Instructions v2 dataset at [DataProvenanceInitiative/niv2_submix_original](https://huggingface.co/datasets/DataProvenanceInitiative/niv2_submix_original). These tasks are: 1. 'task029_winogrande_full_object': Creating a pair of fill in the blank question-answer pairs on objects. 2. 'task030_winogrande_full_person': Creating a pair of fill in the blank questions on persons. 3. 'task031_winogrande_question_generation_object': Writing a fill in the blank question on objects. 4. 'task032_winogrande_question_generation_person': Writing a fill in the blank question on persons. 5. 'task033_winogrande_answer_generation': Answering a fill in the blank question on objects. 6. 'task034_winogrande_question_modification_object': Modifying a fill in the blank question on objects. 7. 'task035_winogrande_question_modification_person': Modifying a fill in the blank question on persons. 8. 'task1391_winogrande_easy_answer_generation': Answering a fill in the blank question on objects. ### Fields - `inputs`: a `string` feature. - `targets`: a `string` feature. - `task_source`: a `string` feature. - `task_name`: a `string` feature. - `template_type`: a `string` feature. ## Citation ``` @inproceedings{wang-etal-2022-super, title = "Super-{N}atural{I}nstructions: Generalization via Declarative Instructions on 1600+ {NLP} Tasks", author = "Wang, Yizhong and Mishra, Swaroop and Alipoormolabashi, Pegah and Kordi, Yeganeh and Mirzaei, Amirreza and Naik, Atharva and Ashok, Arjun and Dhanasekaran, Arut Selvan and Arunkumar, Anjana and Stap, David and Pathak, Eshaan and Karamanolakis, Giannis and Lai, Haizhi and Purohit, Ishan and Mondal, Ishani and Anderson, Jacob and Kuznia, Kirby and Doshi, Krima and Pal, Kuntal Kumar and Patel, Maitreya and Moradshahi, Mehrad and Parmar, Mihir and Purohit, Mirali and Varshney, Neeraj and Kaza, Phani Rohitha and Verma, Pulkit and Puri, Ravsehaj Singh and Karia, Rushang and Doshi, Savan and Sampat, Shailaja Keyur and Mishra, Siddhartha and Reddy A, Sujan and Patro, Sumanta and Dixit, Tanay and Shen, Xudong", editor = "Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue", booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing", month = dec, year = "2022", address = "Abu Dhabi, United Arab Emirates", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.emnlp-main.340", doi = "10.18653/v1/2022.emnlp-main.340", pages = "5085--5109", abstract = "How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions{---}training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones. Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9{\%} on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.", } ```许可证：Apache-2.0 # 自然指令v2温诺格兰德（Winogrande）任务集 - 项目：https://github.com/allenai/natural-instructions - 数据来源：[DataProvenanceInitiative/niv2_submix_original](https://huggingface.co/datasets/DataProvenanceInitiative/niv2_submix_original) ## 数据集详情 本数据集包含收录于[Flan 2022数据集合集](https://github.com/google-research/FLAN/tree/main/flan/v2)中的全部温诺格兰德示例，这些示例最初发表于《Super-Natural-Instructions》研究中。 本数据集的数据源自[DataProvenanceInitiative/niv2_submix_original](https://huggingface.co/datasets/DataProvenanceInitiative/niv2_submix_original)中的预处理版自然指令v2数据集。 本数据集包含以下任务： 1. `task029_winogrande_full_object`：针对物体构建一组填空式问答对。 2. `task030_winogrande_full_person`：针对人物构建一组填空式问题。 3. `task031_winogrande_question_generation_object`：针对物体撰写填空式问题。 4. `task032_winogrande_question_generation_person`：针对人物撰写填空式问题。 5. `task033_winogrande_answer_generation`：针对物体的填空式问题给出答案。 6. `task034_winogrande_question_modification_object`：修改针对物体的填空式问题。 7. `task035_winogrande_question_modification_person`：修改针对人物的填空式问题。 8. `task1391_winogrande_easy_answer_generation`：针对物体的填空式问题给出简易答案。 ### 字段说明 - `inputs`：字符串类型特征。 - `targets`：字符串类型特征。 - `task_source`：字符串类型特征。 - `task_name`：字符串类型特征。 - `template_type`：字符串类型特征。 ## 引用 bibtex @inproceedings{wang-etal-2022-super, title = "Super-{N}atural{I}nstructions: Generalization via Declarative Instructions on 1600+ {NLP} Tasks", author = "Wang, Yizhong and Mishra, Swaroop and Alipoormolabashi, Pegah and Kordi, Yeganeh and Mirzaei, Amirreza and Naik, Atharva and Ashok, Arjun and Dhanasekaran, Arut Selvan and Arunkumar, Anjana and Stap, David and Pathak, Eshaan and Karamanolakis, Giannis and Lai, Haizhi and Purohit, Ishan and Mondal, Ishani and Anderson, Jacob and Kuznia, Kirby and Doshi, Krima and Pal, Kuntal Kumar and Patel, Maitreya and Moradshahi, Mehrad and Parmar, Mihir and Purohit, Mirali and Varshney, Neeraj and Kaza, Phani Rohitha and Verma, Pulkit and Puri, Ravsehaj Singh and Karia, Rushang and Doshi, Savan and Sampat, Shailaja Keyur and Mishra, Siddhartha and Reddy A, Sujan and Patro, Sumanta and Dixit, Tanay and Shen, Xudong", editor = "Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue", booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing", month = dec, year = "2022", address = "Abu Dhabi, United Arab Emirates", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.emnlp-main.340", doi = "10.18653/v1/2022.emnlp-main.340", pages = "5085--5109", abstract = "How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions---training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones. Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9% on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.", }

原始信息汇总

Natural Instructions v2 Winogrande Tasks

数据集详情

该数据集包含所有被包含在Flan 2022 collection中的Winogrande示例，这些示例最初发表在Super-Natural-Instructions中。数据来自预处理的Natural Instructions v2数据集DataProvenanceInitiative/niv2_submix_original。

任务类型

1. task029_winogrande_full_object: 创建关于对象的填空题-答案对。
2. task030_winogrande_full_person: 创建关于人物的填空题。
3. task031_winogrande_question_generation_object: 编写关于对象的填空题。
4. task032_winogrande_question_generation_person: 编写关于人物的填空题。
5. task033_winogrande_answer_generation: 回答关于对象的填空题。
6. task034_winogrande_question_modification_object: 修改关于对象的填空题。
7. task035_winogrande_question_modification_person: 修改关于人物的填空题。
8. task1391_winogrande_easy_answer_generation: 回答关于对象的填空题。

数据字段

• inputs: 字符串特征。
• targets: 字符串特征。
• task_source: 字符串特征。
• task_name: 字符串特征。
• template_type: 字符串特征。

引用

@inproceedings{wang-etal-2022-super, title = "Super-{N}atural{I}nstructions: Generalization via Declarative Instructions on 1600+ {NLP} Tasks", author = "Wang, Yizhong and Mishra, Swaroop and Alipoormolabashi, Pegah and Kordi, Yeganeh and Mirzaei, Amirreza and Naik, Atharva and Ashok, Arjun and Dhanasekaran, Arut Selvan and Arunkumar, Anjana and Stap, David and Pathak, Eshaan and Karamanolakis, Giannis and Lai, Haizhi and Purohit, Ishan and Mondal, Ishani and Anderson, Jacob and Kuznia, Kirby and Doshi, Krima and Pal, Kuntal Kumar and Patel, Maitreya and Moradshahi, Mehrad and Parmar, Mihir and Purohit, Mirali and Varshney, Neeraj and Kaza, Phani Rohitha and Verma, Pulkit and Puri, Ravsehaj Singh and Karia, Rushang and Doshi, Savan and Sampat, Shailaja Keyur and Mishra, Siddhartha and Reddy A, Sujan and Patro, Sumanta and Dixit, Tanay and Shen, Xudong", editor = "Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue", booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing", month = dec, year = "2022", address = "Abu Dhabi, United Arab Emirates", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.emnlp-main.340", doi = "10.18653/v1/2022.emnlp-main.340", pages = "5085--5109", abstract = "How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions{---}training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones. Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9{%} on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.", }

搜集汇总

数据集介绍

构建方式

在自然语言处理领域，指令驱动的任务构建正成为评估模型泛化能力的关键范式。本数据集源自Super-NaturalInstructions项目，其构建过程体现了系统化的任务整合思路：研究者从Flan 2022数据集中精选了Winogrande推理任务，并通过结构化预处理流程，将原始数据转换为统一的指令-响应格式。具体而言，构建者依据任务目标将内容划分为八类专项任务，涵盖对象与人物维度的填空问题生成、答案生成及问题修改等操作，每项任务均通过标准化的字段映射保留原始语义关系，最终形成可被机器学习模型直接解析的规范化语料库。

特点

作为指令学习研究的重要语料，本数据集展现出鲜明的结构化特征与任务多样性。其核心特点在于以细粒度任务分类为基础，通过八个专项任务构建了多层次的语义推理框架，既包含面向对象的填空问题生成，也涉及人物维度的问答修改。数据字段设计兼具简洁性与完整性，inputs与targets字段构成明确的指令-响应对，task_source与task_name字段则确保任务溯源清晰，template_type字段进一步揭示了任务模板的结构化特征。这种设计使得数据集既能支持端到端的指令跟随评估，又能为跨任务泛化研究提供细粒度的分析维度。

使用方法

在实践应用中，本数据集为指令跟随模型的训练与评估提供了标准化基准。研究者可通过HuggingFace平台直接加载数据，利用预定义的inputs字段作为模型输入指令，targets字段作为预期输出目标，构建监督学习流程。针对不同研究需求，既可整合全部任务进行整体性能评估，也可依据task_name字段筛选特定任务类型，开展细粒度的推理能力分析。此外，数据集与Flan系列及Super-NaturalInstructions基准的兼容性，使得其能无缝嵌入现有指令学习研究框架，为模型在常识推理任务上的零样本泛化能力提供量化依据。

背景与挑战

背景概述

在自然语言处理领域，指令驱动的任务泛化研究日益受到关注，旨在探索模型如何依据人类提供的自然语言指令执行未见任务。由Allen Institute for AI等机构于2022年发布的Super-NaturalInstructions数据集，作为这一研究方向的重要基准，整合了超过1600项多样化NLP任务。其中，coref-data/niv2_winogrande_raw作为其子集，专门聚焦于常识推理任务Winogrande的指令化版本，通过将原始填空问题转化为多形式指令任务，如问题生成、答案生成与修改，推动了模型在遵循复杂指令下的推理能力评估。该数据集的构建基于Flan 2022集合与Natural Instructions v2框架，体现了跨任务泛化研究的前沿进展，为指令学习模型的性能测评提供了关键资源。

当前挑战

Winogrande任务的核心挑战在于解决常识推理中的共指消解与上下文依赖问题，要求模型在模糊语境中准确推断实体关系，尤其涉及对象与人物的动态属性区分。数据集的构建过程面临双重困难：一是将原始结构化问题转化为多样化指令任务时，需保持语义一致性并避免引入偏差，例如在问题修改任务中确保逻辑连贯性；二是处理大规模任务集成时的数据溯源与格式标准化，需协调多源数据（如Flan集合与Super-NaturalInstructions）的兼容性，同时维护任务定义的清晰性与评估的可靠性。这些挑战共同考验着指令化数据集的构建精度与泛化效度。

常用场景

经典使用场景

在自然语言处理领域，指令遵循模型的泛化能力评估是核心研究议题之一。coref-data/niv2_winogrande_raw数据集作为Super-NaturalInstructions基准的重要组成部分，其经典使用场景聚焦于通过多样化的填空任务来测试模型对未见指令的理解与执行能力。该数据集整合了Winogrande任务中的对象与人物相关填空问题生成、答案生成及问题修改等子任务，为研究者提供了一个结构化环境，用以衡量模型在遵循复杂自然语言指令时的跨任务泛化表现。

实际应用

在实际应用层面，该数据集支撑了智能助手与对话系统的核心能力开发。基于其填空任务设计的模型训练，能够提升系统在开放域问答中的上下文推理与指代消解精度。例如，在客服自动化或教育辅导场景中，系统可依据用户以自然语言表述的问题生成连贯且准确的填空练习或答案，增强了人机交互的自然性与适应性。

衍生相关工作

围绕该数据集衍生的经典工作主要包括指令遵循模型的架构创新与评估范式的拓展。例如，Tk-Instruct模型利用此类多任务指令数据，在参数规模较小的条件下实现了超越大规模模型的泛化性能。后续研究进一步探索了任务数量、实例密度与模型尺度对泛化效果的影响，催生了如任务元学习、指令压缩等一系列方法论进展，持续丰富了通用自然语言处理模型的研究图谱。