Natural Instructions

Natural-Instructions 项目的目标是为衡量对看不见的任务的泛化提供一个良好的质量基准。这种概括取决于（并受益于）对自然语言指令的理解和推理，这些指令清楚而完整地描述了一项任务（传统上定义为将输入字符串映射到输出字符串）。配备“理解”语言说明的模型应该成功解决任何看不见的任务，如果提供了任务说明的话。
我们已经建立了两个数据集来研究这个目标。我们的 v1.x 数据集利用了现有 NLP 数据集的众包模板。该数据集由 61 个任务组成。 v2.x 数据集建立在早期工作的基础上，具有更简单的模式并包含超过 1.5k 个任务。我们在许多慷慨的社区贡献者的帮助下收集了这些说明。

The goal of the Natural-Instructions project is to provide a high-quality benchmark for measuring generalization to unseen tasks. Such generalization depends on (and benefits from) the understanding and reasoning of natural language instructions, which clearly and comprehensively describe a task (traditionally defined as mapping input strings to output strings). Models equipped with the ability to "understand" language instructions should successfully solve any unseen task when provided with its task description.

We have developed two datasets to investigate this goal. Our v1.x dataset leverages crowdsourced templates from existing NLP datasets, and consists of 61 tasks. The v2.x dataset builds upon prior work, features a simpler schema, and contains over 1.5k tasks. We collected these instructions with the assistance of numerous generous community contributors.

搜集汇总

背景与挑战

背景概述

Natural Instructions数据集旨在评估模型通过理解自然语言指令对未见任务的泛化能力，包含两个版本：v1.x有61个任务，基于众包模板；v2.x扩展至超过1.5k个任务，模式更简单且由社区贡献收集。该数据集由多所研究机构于2022年发布，用于推动跨任务泛化研究。