xsum_factuality

# Dataset Card for XSum Hallucination Annotations

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** [XSUM Hallucination Annotations Homepage](https://research.google/tools/datasets/xsum-hallucination-annotations/)
- **Repository:** [XSUM Hallucination Annotations Homepage](https://github.com/google-research-datasets/xsum_hallucination_annotations)
- **Paper:** [ACL Web](https://www.aclweb.org/anthology/2020.acl-main.173.pdf)
- **Point of Contact:** [[e-mail]](mailto:[e-mail])

### Dataset Summary

Neural abstractive summarization models are highly prone to hallucinate content that is unfaithful to the input document. The popular metric such as ROUGE fails to show the severity of the problem. This dataset contains a large scale human evaluation of several neural abstractive summarization systems to better understand the types of hallucinations they produce. The dataset consists of faithfulness and factuality annotations of abstractive summaries for the XSum dataset. The dataset has crowdsourced 3 judgements for each of 500 x 5 document-system pairs. This will be a valuable resource to the abstractive summarization community.

### Supported Tasks and Leaderboards

* `summarization`: : The dataset can be used to train a model for Summarization,, which consists in summarizing a given document. Success on this task is typically measured by achieving a *high/low* [ROUGE Score](https://huggingface.co/metrics/rouge).

### Languages

The text in the dataset is in English which are abstractive summaries for the [XSum dataset](https://www.aclweb.org/anthology/D18-1206.pdf). The associated BCP-47 code is `en`.

## Dataset Structure

### Data Instances

##### Faithfulness annotations dataset

A typical data point consists of an ID referring to the news article(complete document), summary, and the hallucination span information.

An example from the XSum Faithfulness dataset looks as follows:

```
{
'bbcid': 34687720,
'hallucinated_span_end': 114,
'hallucinated_span_start': 1,
'hallucination_type': 1,
'summary': 'rory mcilroy will take a one-shot lead into the final round of the wgc-hsbc champions after carding a three-under',
'system': 'BERTS2S',
'worker_id': 'wid_0'
}
```

##### Factuality annotations dataset

A typical data point consists of an ID referring to the news article(complete document), summary, and whether the summary is factual or not.

An example from the XSum Factuality dataset looks as follows:

```
{
'bbcid': 29911712,
'is_factual': 0,
'summary': 'more than 50 pupils at a bristol academy have been sent home from school because of a lack of uniform.',
'system': 'BERTS2S',
'worker_id': 'wid_0'
}
```

### Data Fields

##### Faithfulness annotations dataset

Raters are shown the news article and the system summary, and are tasked with identifying and annotating the spans that aren't supported by the input article. The file contains the following columns:

- `bbcid`: Document id in the XSum corpus.
- `system`: Name of neural summarizer.
- `summary`: Summary generated by ‘system’.
- `hallucination_type`: Type of hallucination: intrinsic (0) or extrinsic (1)
- `hallucinated_span`: Hallucinated span in the ‘summary’.
- `hallucinated_span_start`: Index of the start of the hallucinated span.
- `hallucinated_span_end`: Index of the end of the hallucinated span.
- `worker_id`: Worker ID (one of 'wid_0', 'wid_1', 'wid_2')

The `hallucination_type` column has NULL value for some entries which have been replaced iwth `-1`.

##### Factuality annotations dataset

Raters are shown the news article and the hallucinated system summary, and are tasked with assessing the summary whether it is factual or not. The file contains the following columns:

- `bbcid1: Document id in the XSum corpus.
- `system`: Name of neural summarizer.
- `summary`: Summary generated by ‘system’.
- `is_factual`: Yes (1) or No (0)
- `worker_id`: Worker ID (one of 'wid_0', 'wid_1', 'wid_2')

The `is_factual` column has NULL value for some entries which have been replaced iwth `-1`.

### Data Splits

There is only a single split for both the Faithfulness annotations dataset and Factuality annotations dataset.

| | train |
|--------------------------|------:|
| Faithfulness annotations | 11185 |
| Factuality annotations | 5597 |

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/legalcode)

### Citation Information

```
@InProceedings{maynez_acl20,
author = "Joshua Maynez and Shashi Narayan and Bernd Bohnet and Ryan Thomas Mcdonald",
title = "On Faithfulness and Factuality in Abstractive Summarization",
booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
year = "2020",
pages = "1906--1919",
address = "Online",
}
```

### Contributions

Thanks to [@vineeths96](https://github.com/vineeths96) for adding this dataset.

# XSUM幻觉标注数据集卡片

## 目录
- [数据集描述](#dataset-description)
- [数据集概要](#dataset-summary)
- [支持任务与评测基准](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集构建](#dataset-creation)
- [整理初衷](#curation-rationale)
- [源数据](#source-data)
- [标注流程](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差讨论](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集整理者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献](#contributions)

## 数据集描述

- **主页地址**：[XSUM幻觉标注数据集主页](https://research.google/tools/datasets/xsum-hallucination-annotations/)
- **代码仓库**：[XSUM幻觉标注数据集仓库](https://github.com/google-research-datasets/xsum_hallucination_annotations)
- **相关论文**：[ACL官方网站](https://www.aclweb.org/anthology/2020.acl-main.173.pdf)
- **联系方式**：[[e-mail]](mailto:[e-mail])

### 数据集概要
神经抽象式摘要（abstractive summarization）模型极易生成与输入文档不符的幻觉内容。诸如ROUGE评分（ROUGE Score）这类常用指标无法体现该问题的严重程度。本数据集包含针对多款神经抽象式摘要模型的大规模人工评估，以深入剖析其生成的幻觉类型。数据集涵盖针对XSUM数据集（XSum dataset）中抽象式摘要的忠实性与事实性标注。该数据集针对500组文档-系统对，由众包完成了3份标注意见。本资源将为抽象式摘要研究社区提供宝贵支撑。

### 支持任务与评测基准
* `摘要生成`：本数据集可用于训练摘要生成模型，任务目标为对给定文档生成摘要。该任务的性能通常通过*高/低* [ROUGE评分（ROUGE Score）](https://huggingface.co/metrics/rouge) 进行衡量。

### 语言
本数据集文本均为英语，对应XSUM数据集的抽象式摘要，关联BCP-47代码为`en`。

## 数据集结构

### 数据实例

#### 忠实性标注数据集
典型数据点包含指向新闻文章（完整文档）的ID、摘要以及幻觉片段信息。
XSUM忠实性数据集的示例如下：

{
'bbcid': 34687720,
'hallucinated_span_end': 114,
'hallucinated_span_start': 1,
'hallucination_type': 1,
'summary': 'rory mcilroy will take a one-shot lead into the final round of the wgc-hsbc champions after carding a three-under',
'system': 'BERTS2S',
'worker_id': 'wid_0'
}

#### 事实性标注数据集
典型数据点包含指向新闻文章（完整文档）的ID、摘要以及摘要是否符合事实的标注结果。
XSUM事实性数据集的示例如下：

{
'bbcid': 29911712,
'is_factual': 0,
'summary': 'more than 50 pupils at a bristol academy have been sent home from school because of a lack of uniform.',
'system': 'BERTS2S',
'worker_id': 'wid_0'
}

### 数据字段

#### 忠实性标注数据集
标注人员将查看新闻文章与系统生成的摘要，任务为识别并标注输入文档未提供支撑的片段。该文件包含以下列：

- `bbcid`：XSUM语料库中的文档ID
- `system`：神经摘要生成器的名称
- `summary`：由`system`生成的摘要
- `hallucination_type`：幻觉类型：内在型（0）或外在型（1）
- `hallucinated_span`：摘要中的幻觉片段
- `hallucinated_span_start`：幻觉片段的起始索引
- `hallucinated_span_end`：幻觉片段的结束索引
- `worker_id`：标注人员ID（取值为'wid_0'、'wid_1'、'wid_2'之一）

部分条目的`hallucination_type`字段原为空值，现已替换为`-1`。

#### 事实性标注数据集
标注人员将查看新闻文章与系统生成的摘要，任务为评估摘要是否符合事实。该文件包含以下列：

- `bbcid`：XSUM语料库中的文档ID
- `system`：神经摘要生成器的名称
- `summary`：由`system`生成的摘要
- `is_factual`：符合事实（1）或不符合事实（0）
- `worker_id`：标注人员ID（取值为'wid_0'、'wid_1'、'wid_2'之一）

部分条目的`is_factual`字段原为空值，现已替换为`-1`。

### 数据划分
忠实性标注数据集与事实性标注数据集均仅包含单一划分。

| | 训练集 |
|--------------------------|------:|
| 忠实性标注数据集 | 11185 |
| 事实性标注数据集 | 5597 |

## 数据集构建

### 整理初衷
[需要更多信息]

### 源数据

#### 初始数据收集与标准化
[需要更多信息]

#### 源文本生产者是谁？
[需要更多信息]

### 标注流程

#### 标注过程
[需要更多信息]

#### 标注人员是谁？
[需要更多信息]

### 个人与敏感信息
[需要更多信息]

## 数据集使用注意事项

### 数据集的社会影响
[需要更多信息]

### 偏差讨论
[需要更多信息]

### 其他已知局限性
[需要更多信息]

## 附加信息

### 数据集整理者
[需要更多信息]

### 许可信息
[知识共享署名4.0国际许可协议（Creative Commons Attribution 4.0 International）](https://creativecommons.org/licenses/by/4.0/legalcode)

### 引用信息

@InProceedings{maynez_acl20,
author = "Joshua Maynez and Shashi Narayan and Bernd Bohnet and Ryan Thomas Mcdonald",
title = "On Faithfulness and Factuality in Abstractive Summarization",
booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
year = "2020",
pages = "1906--1919",
address = "Online",
}

### 贡献
感谢 [@vineeths96](https://github.com/vineeths96) 添加本数据集。