MRQA 2019

“2019 MRQA 共享任务侧重于泛化。我们发布了一个官方训练数据集，其中包含来自现有 QA 数据集的示例，并在十个隐藏的 QA 测试数据集上评估提交的模型。训练和测试数据集可能在以下一些方面有所不同：段落分布：测试示例可能涉及来自不同来源（例如，科学、新闻、小说、医学摘要等）的段落，具有明显的句法和词汇差异。问题分布：测试示例可能强调不同风格的问题（例如，以实体为中心，相关的，其他重新表述为 QA 的任务等）可能来自不同的来源（例如，众包工作者、领域专家、考试作者等） 联合分布：测试示例可能会根据问题与文章的关系而有所不同（例如，收集独立与依赖证据、多跳等）"

The 2019 MRQA Shared Task focuses on generalization. We have released an official training dataset containing examples sourced from existing QA datasets, and submitted models are evaluated on ten hidden QA test datasets. Training and test datasets may differ in several aspects:
1. Passage distribution: Test instances may involve passages from diverse sources (e.g., scientific literature, news articles, fiction, medical abstracts, etc.) with notable syntactic and lexical disparities.
2. Question distribution: Test instances may highlight questions of various styles (e.g., entity-centric, relational, and other tasks reformulated as QA) and may stem from different origins (e.g., crowdworkers, domain experts, exam authors, etc.).
3. Joint distribution: Test instances may vary according to the relationship between questions and passages (e.g., requiring standalone vs. supporting evidence, multi-hop reasoning, etc.).

搜集汇总

背景与挑战

背景概述

MRQA 2019是一个专注于机器阅读理解泛化评估的数据集，包含一个官方训练数据集和十个隐藏的测试数据集，测试集在段落来源、问题风格及问题与文章关系上可能呈现多样化差异。该数据集由斯坦福大学等多所研究机构于2019年联合发布。