mtc/span_absinth_german_faithfulness_detection_dataset

---
dataset_info:
features:
- name: id
dtype: int64
- name: text
dtype: string
- name: article_id
dtype: int64
- name: system
dtype: string
- name: sentence_ord
dtype: int64
- name: Comments
sequence: string
- name: pre_context
dtype: string
- name: post_context
dtype: string
- name: label
dtype: string
splits:
- name: test
num_bytes: 563651
num_examples: 1125
- name: train
num_bytes: 1150346
num_examples: 2608
- name: validation
num_bytes: 96421
num_examples: 200
download_size: 749137
dataset_size: 1810418
configs:
- config_name: default
data_files:
- split: test
path: data/test-*
- split: train
path: data/train-*
- split: validation
path: data/validation-*
license: apache-2.0
task_categories:
- text-classification
language:
- de
tags:
- croissant
size_categories:
- 1K<n<10K
---

# Dataset Card for "Span Absinth - Hallucination Detection Dataset of German News Summarization"

## Dataset Description

Span Absinth is an extension of the [Absinth](https://huggingface.co/datasets/mtc/absinth_german_faithfulness_detection_dataset) dataset, where each hallucinated summary-sentence has been augmented with span annotations, that define which part of the sentence is hallucinated. Span annotations have the advantage of
effectively isolating hallucinations at the token level.

Please refer to our [paper](https://arxiv.org/abs/2403.03750) and [Absinth](https://huggingface.co/datasets/mtc/absinth_german_faithfulness_detection_dataset), for more details about the dataset.

**Important:**
The test set contains instances that share source articles with the train set and validation set.
To eliminate test instances with these overlapping source articles, you can remove rows with article_id values: **[131, 139, 15, 16, 146, 151, 35, 163, 41, 175, 177, 179, 181, 185, 187, 60, 61, 70, 87, 96, 112, 82]**

## Dataset Structure

The dataset is almost identical to the original Absinth dataset, except the _label_ column, which will be explained in detail:
**label**: str
- _Faithful_: The entire summary-sentence is faithful to the article.
- For hallucinated samples, the label contains a list of dictionaries containing information about the span:
- _start_: int - The start index of the span relative to the original summary-sentence.
- _end_: int - The end index of the span relative to the original summary-sentence.
- _span_: str - The hallucinated span text.
- _span_label_: str - The span label, can be either _Intrinsic_ or _Extrinsic_.

### Citation Information

```
@inproceedings{mascarell-etal-2024-german,
title = "German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset",
author = "Mascarell, Laura and
Chalummattu, Ribin and
Rios, Annette",
booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)",
month = May,
year = "2024",
address = "Turin, Italy",
publisher = "",
url = "",
pages = "",
abstract = "The advent of Large Language Models (LLMs) has lead to remarkable progress on a wide range of natural language processing tasks. Despite the advances, these large-sized models still suffer from hallucinating information in their output, which poses a major issue in automatic text summarization, as we must guarantee that the generated summary is consistent with the content of the source document. Previous research addresses the challenging task of detecting hallucinations in the output (i.e. inconsistency detection) in order to evaluate the faithfulness of the generated summaries. However, these works primarily focus on English and recent multilingual approaches lack German data. This work presents absinth, a manually annotated dataset for hallucination detection in German news summarization and explores the capabilities of novel open-source LLMs on this task in both fine-tuning and in-context learning settings. We open-source and release the absinth dataset to foster further research on hallucination detection in German.",
}
```

原始信息汇总

数据集概述

数据集名称

Span Absinth - Hallucination Detection Dataset of German News Summarization

数据集特征

• id: int64
• text: string
• article_id: int64
• system: string
• sentence_ord: int64
• Comments: sequence: string
• pre_context: string
• post_context: string
• label: string

数据集分割

• test: 1125 examples, 563651 bytes
• train: 2608 examples, 1150346 bytes
• validation: 200 examples, 96421 bytes

数据集大小

• 下载大小: 749137 bytes
• 数据集大小: 1810418 bytes

数据集配置

• config_name: default
• data_files:
  • test: data/test-*
  • train: data/train-*
  • validation: data/validation-*

许可证

apache-2.0

任务类别

• text-classification

语言

• de

标签说明

• label: str
  • Faithful: 整个摘要句子忠实于文章。
  • 对于幻觉样本，标签包含一个字典列表，包含关于跨度的信息：
    • start: int - 跨度相对于原始摘要句子的起始索引。
    • end: int - 跨度相对于原始摘要句子的结束索引。
    • span: str - 幻觉跨度文本。
    • span_label: str - 跨度标签，可以是 Intrinsic 或 Extrinsic。

重要提示

测试集包含与训练集和验证集共享源文章的实例。为消除这些重叠的源文章，可以删除具有以下 article_id 值的行：[131, 139, 15, 16, 146, 151, 35, 163, 41, 175, 177, 179, 181, 185, 187, 60, 61, 70, 87, 96, 112, 82]