multi_re_qa

# Dataset Card for MultiReQA

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://github.com/google-research-datasets/MultiReQA
- **Repository:** https://github.com/google-research-datasets/MultiReQA
- **Paper:** https://arxiv.org/pdf/2005.02507.pdf
- **Leaderboard:**
- **Point of Contact:**

### Dataset Summary
MultiReQA contains the sentence boundary annotation from eight publicly available QA datasets including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, and TextbookQA. Five of these datasets, including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, contain both training and test data, and three, in cluding BioASQ, RelationExtraction, TextbookQA, contain only the test data (also includes DuoRC but not specified in the official documentation)
### Supported Tasks and Leaderboards

- Question answering (QA)
- Retrieval question answering (ReQA)
### Languages

Sentence boundary annotation for SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, TextbookQA and DuoRC

## Dataset Structure

### Data Instances

The general format is:
`
{
"candidate_id": <candidate_id>,
"response_start": <response_start>,
"response_end": <response_end>
}
...
`

An example from SearchQA:
`{'candidate_id': 'SearchQA_000077f3912049dfb4511db271697bad/_0_1',
'response_end': 306,
'response_start': 243} `

### Data Fields

`
{
"candidate_id": <STRING>,
"response_start": <INT>,
"response_end": <INT>
}
...
`
- **candidate_id:** The candidate id of the candidate sentence. It consists of the original qid from the MRQA shared task.
- **response_start:** The start index of the sentence with respect to its original context.
- **response_end:** The end index of the sentence with respect to its original context

### Data Splits

Train and Dev splits are available only for the following datasets,
- SearchQA
- TriviaQA
- HotpotQA
- SQuAD
- NaturalQuestions

Test splits are available only for the following datasets,
- BioASQ
- RelationExtraction
- TextbookQA

The number of candidate sentences for each dataset in the table below.

| | MultiReQA | |
|--------------------|-----------|---------|
| | train | test |
| SearchQA | 629,160 | 454,836 |
| TriviaQA | 335,659 | 238,339 |
| HotpotQA | 104,973 | 52,191 |
| SQuAD | 87,133 | 10,642 |
| NaturalQuestions | 106,521 | 22,118 |
| BioASQ | - | 14,158 |
| RelationExtraction | - | 3,301 |
| TextbookQA | - | 3,701 |

## Dataset Creation

### Curation Rationale

MultiReQA is a new multi-domain ReQA evaluation suite composed of eight retrieval QA tasks drawn from publicly available QA datasets from the [MRQA shared task](https://mrqa.github.io/). The dataset was curated by converting existing QA datasets from [MRQA shared task](https://mrqa.github.io/) to the format of MultiReQA benchmark.
### Source Data

#### Initial Data Collection and Normalization

The Initial data collection was performed by converting existing QA datasets from MRQA shared task to the format of MultiReQA benchmark.
#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

The annotators/curators of the dataset are [mandyguo-xyguo](https://github.com/mandyguo-xyguo) and [mwurts4google](https://github.com/mwurts4google), the contributors of the official MultiReQA github repository
### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

The annotators/curators of the dataset are [mandyguo-xyguo](https://github.com/mandyguo-xyguo) and [mwurts4google](https://github.com/mwurts4google), the contributors of the official MultiReQA github repository

### Licensing Information

[More Information Needed]

### Citation Information

```
@misc{m2020multireqa,
title={MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models},
author={Mandy Guo and Yinfei Yang and Daniel Cer and Qinlan Shen and Noah Constant},
year={2020},
eprint={2005.02507},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

### Contributions

Thanks to [@Karthik-Bhaskar](https://github.com/Karthik-Bhaskar) for adding this dataset.

# 多领域检索问答(MultiReQA)数据集卡片

## 目录
- [数据集说明](#dataset-description)
- [数据集摘要](#dataset-summary)
- [支持任务与评测榜单](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集构建](#dataset-creation)
- [构建初衷](#curation-rationale)
- [源数据](#source-data)
- [标注信息](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差分析](#discussion-of-biases)
- [其他已知局限](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集维护者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献致谢](#contributions)

## 数据集说明
- **主页:** https://github.com/google-research-datasets/MultiReQA
- **代码仓库:** https://github.com/google-research-datasets/MultiReQA
- **论文:** https://arxiv.org/pdf/2005.02507.pdf
- **评测榜单:**
- **联系方式:**

### 数据集摘要
MultiReQA包含来自8个公开可用问答数据集的句子边界标注，涵盖SearchQA、TriviaQA、HotpotQA、NaturalQuestions、SQuAD、BioASQ、关系抽取(RelationExtraction)以及TextbookQA。其中5个数据集包含训练集与测试集，具体为SearchQA、TriviaQA、HotpotQA、NaturalQuestions与SQuAD；剩余3个数据集仅包含测试集，即BioASQ、关系抽取(RelationExtraction)、TextbookQA（此外还包含DuoRC，但官方文档未明确说明）。

### 支持任务与评测榜单
- 问答(Question Answering, QA)
- 检索式问答(Retrieval Question Answering, ReQA)

### 语言
为SearchQA、TriviaQA、HotpotQA、NaturalQuestions、SQuAD、BioASQ、RelationExtraction、TextbookQA以及DuoRC提供句子边界标注。

## 数据集结构

### 数据实例
通用格式如下：

{
"candidate_id": <candidate_id>,
"response_start": <response_start>,
"response_end": <response_end>
}
...

以下为来自SearchQA的示例：

{'candidate_id': 'SearchQA_000077f3912049dfb4511db271697bad/_0_1',
'response_end': 306,
'response_start': 243}

### 数据字段

{
"candidate_id": <STRING>,
"response_start": <INT>,
"response_end": <INT>
}
...

- **候选ID(candidate_id):** 候选句子的唯一标识，由MRQA共享任务(MRQA shared task)中的原始qid构成。
- **回答起始位置(response_start):** 该句子相对于其原始上下文的起始索引。
- **回答结束位置(response_end):** 该句子相对于其原始上下文的结束索引。

### 数据划分
训练集与开发集仅在以下数据集可用：
- SearchQA
- TriviaQA
- HotpotQA
- SQuAD
- NaturalQuestions

测试集仅在以下数据集可用：
- BioASQ
- RelationExtraction
- TextbookQA

各数据集的候选句子数量如下表所示：

| | MultiReQA | |
|--------------------|-----------|---------|
| | 训练集 | 测试集 |
| SearchQA | 629,160 | 454,836 |
| TriviaQA | 335,659 | 238,339 |
| HotpotQA | 104,973 | 52,191 |
| SQuAD | 87,133 | 10,642 |
| NaturalQuestions | 106,521 | 22,118 |
| BioASQ | - | 14,158 |
| RelationExtraction | - | 3,301 |
| TextbookQA | - | 3,701 |

## 数据集构建

### 构建初衷
MultiReQA是一套全新的多领域检索式问答评估套件，由来自[MRQA共享任务(MRQA shared task)](https://mrqa.github.io/)的公开可用问答数据集整合而成。本数据集通过将MRQA共享任务中的现有问答数据集转换为MultiReQA基准格式完成构建。

### 源数据
#### 初始数据收集与标准化
初始数据收集与标准化工作通过将MRQA共享任务中的现有问答数据集转换为MultiReQA基准格式完成。
#### 源语言生产者信息
[需要更多信息]

### 标注信息
#### 标注流程
[需要更多信息]
#### 标注者信息
本数据集的标注者/维护者为[mandyguo-xyguo](https://github.com/mandyguo-xyguo)与[mwurts4google](https://github.com/mwurts4google)，即官方MultiReQA GitHub仓库的贡献者。

### 个人与敏感信息
[需要更多信息]

## 数据集使用注意事项

### 数据集的社会影响
[需要更多信息]

### 偏差分析
[需要更多信息]

### 其他已知局限
[需要更多信息]

## 附加信息

### 数据集维护者
本数据集的标注者/维护者为[mandyguo-xyguo](https://github.com/mandyguo-xyguo)与[mwurts4google](https://github.com/mwurts4google)，即官方MultiReQA GitHub仓库的贡献者。

### 许可信息
[需要更多信息]

### 引用信息

@misc{m2020multireqa,
title={MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models},
author={Mandy Guo and Yinfei Yang and Daniel Cer and Qinlan Shen and Noah Constant},
year={2020},
eprint={2005.02507},
archivePrefix={arXiv},
primaryClass={cs.CL}
}

（注：原论文标题可译为"MultiReQA：面向检索式问答模型的跨领域评测基准"）

### 贡献致谢
感谢[@Karthik-Bhaskar](https://github.com/Karthik-Bhaskar)添加本数据集。