summexecedit

# Factual Consistency in Summarization

Evaluate your model's ability to detect and explain the factual inconsistency in summaries. This repo contains the benchmark from our paper ["SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits"](https://arxiv.org/abs/2412.13378).

## SummExecEdit Benchmark

This benchmark is built over our previous benchmark - [SummEdits](https://huggingface.co/datasets/Salesforce/summedits). Consistent summaries are used from SummEdits. New inconsistent and challenging summaries are generated using executable editing mechanism.

We release the 4,241 samples of data for the 10 domains in the SummExecEdit. Each sample has entries for:
- `sample_id`: unique ID for the sample,
- `doc_id`: unique ID for the document,
- `doc`: input document,
- `original_summary`: the summary that is either consistent or inconsistent with the facts in the document,
- `original_text`: the text in original_summary to be replaced to introduce factual inconsistency,
- `replace_text`: the text with which original_text is replaced that introduces factual inconsistency,
- `edited_summary`: the summary that is either consistent or inconsistent with the facts in the document,
- `explanation`: explanation for factual inconsistency if present,
- `domain`: domain to which document and summary belongs,
- `model`: model which is used for executable editing i.e. generating original_text, replace_text, and explanation,
- `edit_type`: "summedits" if the summary is factually consistent, and summexecedit otherwise,

If you find this useful, please consider citing:

```bibtex
@misc{thorat2024summexeceditfactualconsistencybenchmark,
title={SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits},
author={Onkar Thorat and Philippe Laban and Chien-Sheng Wu},
year={2024},
eprint={2412.13378},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.13378},
}
```

# 摘要任务中的事实一致性
评估模型检测并解释摘要中事实不一致性的能力。本仓库收录了我们在论文《SummExecEdit：一种基于可执行编辑的摘要事实一致性基准测试》（SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits）中提出的基准数据集。

## SummExecEdit 基准数据集
本基准数据集构建于我们此前推出的基准数据集SummEdits（SummEdits）之上，该数据集可通过如下链接获取：https://huggingface.co/datasets/Salesforce/summedits。我们从SummEdits中选取了符合事实一致性的摘要样本，并通过可执行编辑机制生成了全新的存在事实不一致性且更具挑战性的新增摘要样本。

我们在SummExecEdit基准数据集中发布了覆盖10个领域的4241条数据样本，每条样本包含以下字段：
- `sample_id`：样本唯一标识符
- `doc_id`：源文档唯一标识符
- `doc`：输入源文档
- `original_summary`：与源文档事实既可能一致也可能不一致的原始摘要
- `original_text`：原始摘要中用于引入事实不一致性的待替换文本片段
- `replace_text`：用于替换`original_text`的、会引入事实不一致性的文本片段
- `edited_summary`：与源文档事实既可能一致也可能不一致的编辑后摘要
- `explanation`：若存在事实不一致性，则为对应的解释文本
- `domain`：源文档与摘要所属的领域
- `model`：用于执行可编辑操作（即生成`original_text`、`replace_text`及`explanation`）的模型
- `edit_type`：若摘要事实一致则为`summedits`，否则为`summexecedit`

若您认为本数据集对研究有所助益，请引用如下文献：
bibtex
@misc{thorat2024summexeceditfactualconsistencybenchmark,
title={SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits},
author={Onkar Thorat and Philippe Laban and Chien-Sheng Wu},
year={2024},
eprint={2412.13378},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.13378},
}