disfl_qa

# Dataset Card for DISFL-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** [Disfl-QA](https://github.com/google-research-datasets/disfl-qa)
- **Paper:** [Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering](https://arxiv.org/pdf/2106.04016.pdf)
- **Point of Contact:** [disfl-qa team]([e-mail])

### Dataset Summary

Disfl-QA is a targeted dataset for contextual disfluencies in an information seeking setting, namely question answering over Wikipedia passages. Disfl-QA builds upon the SQuAD-v2 ([Rajpurkar et al., 2018](https://www.aclweb.org/anthology/P18-2124/)) dataset, where each question in the dev set is annotated to add a contextual disfluency using the paragraph as a source of distractors.

The final dataset consists of ~12k (disfluent question, answer) pairs. Over 90\% of the disfluencies are corrections or restarts, making it a much harder test set for disfluency correction. Disfl-QA aims to fill a major gap between speech and NLP research community. The authors hope the dataset can serve as a benchmark dataset for testing robustness of models against disfluent inputs.

The expriments reveal that the state-of-the-art models are brittle when subjected to disfluent inputs from Disfl-QA. Detailed experiments and analyses can be found in the [paper](https://arxiv.org/pdf/2106.04016.pdf).

### Supported Tasks and Leaderboards

[More Information Needed]

### Languages

The dataset is in English only.

## Dataset Structure

### Data Instances

This example was too long and was cropped:
```
{
"answers": {
"answer_start": [94, 87, 94, 94],
"text": ["10th and 11th centuries", "in the 10th and 11th centuries", "10th and 11th centuries", "10th and 11th centuries"]
},
"context": "\"The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave thei...",
"id": "56ddde6b9a695914005b9629",
"original question": "When were the Normans in Normandy?",
"disfluent question": "From which countries no tell me when were the Normans in Normandy?"
"title": "Normans"
}
```
### Data Fields

- `id`: a `string` feature.
- `title`: a `string` feature.
- `context`: a `string` feature.
- `original question`: Original question from SQuAD-v2 (a `string` feature)
- `disfluent question`: Disfluent question from Disfl-QA (a `string` feature)
- `answers`: a dictionary feature containing:
- `text`: a `string` feature.
- `answer_start`: a `int32` feature.

### Data Splits

Disfl-QA consists of ~12k disfluent questions with the following train/dev/test splits:
| File | Questions |
|-----|-----|
|train.json | 7182 |
|dev.json | 1000 |
|test.json | 3643 |

## Dataset Creation

### Curation Rationale

The research in NLP and speech community has been impeded by the lack of curated datasets containing such disfluencies. The datasets available today are mostly conversational in nature, and span a limited number of very specific domains (e.g., telephone conversations, court proceedings). Furthermore, only a small fraction of the utterances in these datasets contain disfluencies, with a limited and skewed distribution of disfluencies types. In the most popular dataset in the literature, the SWITCHBOARD corpus (Godfrey et al., 1992), only 5.9% of the words are disfluencies (Charniak and Johnson, 2001), of which > 50% are repetitions (Shriberg, 1996), which has been shown to be the relatively simpler form of disfluencies (Zayats et al., 2014; Jamshid Lou et al., 2018; Zayats et al., 2019). To fill this gap, the authors presented DISFL-QA, the first dataset containing contextual disfluencies in an information seeking setting, namely question answering over Wikipedia passages.

### Source Data

#### Initial Data Collection and Normalization

DISFL-QA is constructed by asking human raters to insert disfluencies in questions from SQUAD-v2, a popular question answering dataset, using the passage and remaining questions as context. These contextual disfluencies lend naturalness to DISFL-QA, and challenge models relying on shallow matching between question and context to predict an answer.

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

Each question associated with the paragraph is sent for a human annotation task to add a contextual disfluency using the paragraph as a source of distractors. Finally, to ensure the quality of the dataset, a subsequent round of human evaluation with an option to re-annotate is conducted.

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

Disfl-QA dataset is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).

### Citation Information

```
@inproceedings{gupta-etal-2021-disflqa,
title = "{Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering}",
author = "Gupta, Aditya and Xu, Jiacheng and Upadhyay, Shyam and Yang, Diyi and Faruqui, Manaal",
booktitle = "Findings of ACL",
year = "2021"
}
```

### Contributions

Thanks to [@bhavitvyamalik](https://github.com/bhavitvyamalik) for adding this dataset.

# DISFL-QA 数据集卡片：面向问答场景中语言不流畅现象理解的基准数据集

## 目录
- [目录](#table-of-contents)
- [数据集描述](#dataset-description)
- [数据集概述](#dataset-summary)
- [支持任务与基准榜单](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集构建](#dataset-creation)
- [构建初衷](#curation-rationale)
- [源数据](#source-data)
- [标注流程](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏倚性讨论](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集维护者](#dataset-curators)
- [授权信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献者](#contributions)

## 数据集描述

- **主页**：[Disfl-QA](https://github.com/google-research-datasets/disfl-qa)
- **论文**：[Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering](https://arxiv.org/pdf/2106.04016.pdf)
- **联系方式**：[disfl-qa 团队]([e-mail])

### 数据集概述

DISFL-QA是针对信息检索场景（即基于维基百科篇章的问答任务）中的上下文语言不流畅现象(Disfluencies)构建的专用数据集。该数据集基于SQuAD-v2（[Rajpurkar等人，2018](https://www.aclweb.org/anthology/P18-2124/)）数据集构建，其开发集中的每道问题均被标注，以篇章作为干扰源，添加上下文语言不流畅现象。

最终数据集包含约1.2万组（不流畅问题、答案）配对。其中超过90%的语言不流畅现象属于修正或重启类型，这使得该数据集成为语言不流畅现象修正任务中难度更高的测试集。DISFL-QA旨在填补语音与自然语言处理（NLP）研究领域之间的重大空白。作者期望该数据集可作为基准数据集，用于测试模型面对不流畅输入时的鲁棒性。

实验结果表明，当前最先进的模型在面对DISFL-QA数据集的不流畅输入时表现脆弱。详细的实验与分析可参考该[论文](https://arxiv.org/pdf/2106.04016.pdf)。

### 支持任务与基准榜单
[需补充更多信息]

### 语言
本数据集仅包含英文内容。

## 数据集结构

### 数据实例
本示例因过长已被裁剪：
json
{
"answers": {
"answer_start": [94, 87, 94, 94],
"text": ["10th and 11th centuries", "in the 10th and 11th centuries", "10th and 11th centuries", "10th and 11th centuries"]
},
"context": ""The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave thei...",
"id": "56ddde6b9a695914005b9629",
"original question": "When were the Normans in Normandy?",
"disfluent question": "From which countries no tell me when were the Normans in Normandy?",
"title": "Normans"
}

### 数据字段
- `id`：字符串类型特征。
- `title`：字符串类型特征。
- `context`：字符串类型特征。
- `original question`：源自SQuAD-v2的原始问题（字符串类型特征）
- `disfluent question`：DISFL-QA中的不流畅问题（字符串类型特征）
- `answers`：包含以下字段的字典类型特征：
- `text`：字符串类型特征。
- `answer_start`：int32类型特征。

### 数据划分
DISFL-QA包含约1.2万道不流畅问题，其训练/开发/测试划分如下：
| 文件名 | 问题数量 |
| :---------- | :--------- |
| train.json | 7182 |
| dev.json | 1000 |
| test.json | 3643 |

## 数据集构建

### 构建初衷
当前自然语言处理与语音研究领域因缺乏包含此类语言不流畅现象的高质量标注数据集而受到阻碍。现有的相关数据集大多以对话场景为核心，且仅覆盖少数特定领域（例如电话通话、法庭庭审）。此外，这些数据集中仅有极小比例的话语包含语言不流畅现象，且不流畅现象的类型分布有限且偏倚。在学术界最常用的SWITCHBOARD语料库（Godfrey等人，1992）中，仅有5.9%的词汇属于不流畅现象（Charniak与Johnson，2001），其中超过50%为重复现象（Shriberg，1996）——这类不流畅现象已被证实属于相对简单的类型（Zayats等人，2014；Jamshid Lou等人，2018；Zayats等人，2019）。为填补这一空白，作者团队推出了DISFL-QA，这是首个面向信息检索场景（即基于维基百科篇章的问答任务）的上下文语言不流畅现象数据集。

### 源数据
#### 初始数据收集与规范化
DISFL-QA的构建方式为：邀请人类标注员以维基百科篇章与其余问题作为上下文，在热门问答数据集SQuAD-v2的问题中添加上下文语言不流畅现象。这类上下文不流畅现象赋予了DISFL-QA自然性，同时对依赖问题与篇章浅层匹配来预测答案的模型构成挑战。

#### 源语言生产者是谁？
[需补充更多信息]

### 标注流程
#### 标注过程
每道与篇章绑定的问题都会被分配至人类标注任务，要求标注员以篇章作为干扰源，为其添加上下文语言不流畅现象。最后，为确保数据集质量，团队会开展第二轮人类评估，并允许标注员重新进行标注。

#### 标注员是谁？
[需补充更多信息]

### 个人与敏感信息
[需补充更多信息]

## 数据集使用注意事项

### 数据集的社会影响
[需补充更多信息]

### 偏倚性讨论
[需补充更多信息]

### 其他已知局限性
[需补充更多信息]

## 附加信息

### 数据集维护者
[需补充更多信息]

### 授权信息
DISFL-QA数据集采用[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)授权协议。

### 引用信息
bibtex
@inproceedings{gupta-etal-2021-disflqa,
title = "{Disfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering}",
author = "Gupta, Aditya and Xu, Jiacheng and Upadhyay, Shyam and Yang, Diyi and Faruqui, Manaal",
booktitle = "Findings of ACL",
year = "2021"
}

### 贡献者
感谢[@bhavitvyamalik](https://github.com/bhavitvyamalik)为本数据集添加至基准库。