MultiReQA

我们正在创建一个具有挑战性的新基准 MultiReQA：检索问答模型的跨域评估。检索问答 (ReQA) 是从开放语料库中检索问题的句子级答案的任务。 MultiReQA 是一个新的多域 ReQA 评估套件，由 8 个检索 QA 任务组成，这些任务来自 MRQA 共享任务的公开可用的 QA 数据集。我们认为 MultiReQA 测试检索 QA 模型执行域迁移任务的能力。 该存储库托管将现有 QA 数据集从 MRQA 共享任务转换为 MultiReQA 基准格式的代码，以及 QA 数据集的句子边界注释以准确重现我们的工作。请注意，我们不会重新分配 MRQA 共享任务上可用的原始数据集中的内容，而只是重新分配句子边界注释。We are introducing a challenging new benchmark, MultiReQA: a cross-domain evaluation for retrieval-based question answering (ReQA) models. Retrieval-based Question Answering (ReQA) refers to the task of retrieving sentence-level answers to questions from an open corpus. MultiReQA is a novel multi-domain ReQA evaluation suite consisting of 8 retrieval QA tasks sourced from publicly available QA datasets of the MRQA Shared Task. We posit that MultiReQA tests the capability of ReQA models to perform domain adaptation tasks. This repository hosts code that converts existing QA datasets from the MRQA Shared Task into the MultiReQA benchmark format, alongside sentence boundary annotations for the QA datasets to enable accurate reproduction of our work. Please note that we do not redistribute the content from the original datasets available via the MRQA Shared Task; rather, we only redistribute the sentence boundary annotations.

搜集汇总

背景与挑战

背景概述

MultiReQA是一个用于评估检索问答模型跨域性能的基准套件，包含来自MRQA共享任务的8个任务，并提供了句子边界注释。该数据集由卡内基梅隆大学和Google Research于2020年发布，旨在测试模型在域迁移任务中的能力。