Babelscape/story-summeval

---
language:
- en
license:
- cc-by-nc-sa-4.0
pretty_name: Story SummEval
size_categories:
- n<1K
---
# Dataset Card for Story-SummEval

## Dataset Description
For a thorough description of the data creation please refer to the ACL 2024 paper:
"FENICE: Factuality Evaluation of summarization based on NLI and Claim Extraction", Scirè et al. (2024).
### Summary
This dataset contains summaries of stories from Gutenberg and Wikisource along with their factuality labels.
Summaries are generated from several models provided by the paper "Echoes from Alexandria" by Scirè et al. (2023).

### Composition
- **Number of (summary, label) pairs**: 319
- **Sources**:
- Gutenberg
- Wikisource

### Dataset Structure
Each entry in the dataset includes:
- `summary`: The summary of the story.
- `label`: The factuality label of the summary.
- `text_id`: Identifier for the original story text.
- `source`: The source of the story text (either 'gutenberg' or 'wikisource').

To retrieve the story texts:
- If the source is 'gutenberg', match the `text_id` value with the `id` column in the [manu/project_gutenberg](https://huggingface.co/datasets/manu/project_gutenberg) dataset using the 'en' split.
- If the source is 'wikisource', match the `text_id` value with the `title` column in the [wikimedia/wikisource](https://huggingface.co/datasets/wikimedia/wikisource) dataset using the '20231201.en' split.

## Citation Information

```bibtex
@inproceedings{scire-etal-2024-fenice,
title = "{FENICE}: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction",
author = "Scir{\`e}, Alessandro and Ghonim, Karim and Navigli, Roberto",
editor = "Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek",
booktitle = "Findings of the Association for Computational Linguistics ACL 2024",
month = aug,
year = "2024",
address = "Bangkok, Thailand and virtual meeting",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-acl.841",
pages = "14148--14161",
}
```