Helsinki-NLP/shroom-cap

---
license: mpl-2.0
tags:
- multilingual
- hallucination-detection
- scientific-text
- cross-lingual
- classification
- factuality
- fluency
- LLM-evaluation
---

# SHROOM-CAP: Shared Task on Hallucinations and Related Observable Overgeneration Mistakes in Crosslingual Analyses of Publications

## Dataset Summary
SHROOM-CAP is a multilingual dataset for hallucination detection in scientific text generated by large language models (LLMs). The dataset covers nine languages: five high-resource languages (English, French, Hindi, Italian, and Spanish) and four low-resource Indic languages (Bengali, Gujarati, Malayalam, and Telugu). Each instance consists of LLM-generated text, token sequences, logits, and metadata about the source scientific publication. The dataset provides binary labels for:

- **Factual mistakes:** whether the text contains hallucinated or factually incorrect content.
- **Fluency mistakes:** whether the text contains linguistic errors affecting readability.

The task frames hallucination detection as a binary classification problem, with LLMs required to predict factual and fluency mistakes.

## Dataset Structure
The dataset is organized into the following splits:

| Split | Examples | Description |
|-------|---------|------------|
| `train` | 1,755 | Training set batch 1 (en, hi, es, fr, it) |
| `validation` | 1,200 | Validation set (en, hi, es, fr, it) |
| `test` | 4,384 | Test set (all 9 languages, including IndicLanguages bn, te, ml, gu), labels not included to help fight against leakage. Contact the authors for more info. |

Each example contains:
- `index`: unique identifier
- `title`, `abstract`, `doi`, `url`, `datafile`: source publication metadata
- `authors`: list of author names (`first` and `last`)
- `question`: question about the publication
- `model_id`: the LLM used for generation
- `model_config`: model configuration parameters
- `prompt`: prompt used for generation
- `output_text`: LLM-generated answer
- `output_tokens`: tokenized model output
- `output_logits`: token-level logits
- `has_fluency_mistakes`: binary label (`y`/`n`) or `null` for test
- `has_factual_mistakes`: binary label (`y`/`n`) or `null` for test

## Source
- Sinha, Aman et al. (2025). [SHROOM-CAP: Shared Task on Hallucinations and Related Observable Overgeneration Mistakes in Crosslingual Analyses of Publications](https://aclanthology.org/2025.chomps-main.7/). *Proceedings of CHOMPS 2025*.

## Citation
```bibtex
@inproceedings{sinha-etal-2025-shroom,
title = "{SHROOM}-{CAP}: Shared Task on Hallucinations and Related Observable Overgeneration Mistakes in Crosslingual Analyses of Publications",
author = "Sinha, Aman and
Gamba, Federica and
V{\'a}zquez, Ra{\'u}l and
Mickus, Timothee and
Chattopadhyay, Ahana and
Zanella, Laura and
Arakkal Remesh, Binesh and
Kankanampati, Yash and
Chandramania, Aryan and
Agarwal, Rohit",
editor = {Sinha, Aman and
V{\'a}zquez, Ra{\'u}l and
Mickus, Timothee and
Agarwal, Rohit and
Buhnila, Ioana and
Schmidtov{\'a}, Patr{\'i}cia and
Gamba, Federica and
Prasad, Dilip K. and
Tiedemann, J{\"o}rg},
booktitle = "Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025)",
month = dec,
year = "2025",
address = "Mumbai, India",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.chomps-main.7/",
pages = "70--80",
ISBN = "979-8-89176-308-1",
}

license: mpl-2.0
tags:
- 多语言(multilingual)
- 幻觉检测(hallucination detection)
- 科学文本(scientific-text)
- 跨语言(cross-lingual)
- 分类(classification)
- 事实性(factuality)
- 流畅性(fluency)
- 大语言模型评估(LLM-evaluation)

# SHROOM-CAP：跨语言出版物分析中的幻觉与相关可观测生成过度错误共享任务

## 数据集概述
SHROOM-CAP是一款用于检测大语言模型(LLM)生成的科学文本中幻觉的多语言数据集。该数据集涵盖9种语言：5种高资源语言（英语、法语、印地语、意大利语、西班牙语）以及4种低资源印度语言（孟加拉语、古吉拉特语、马拉雅拉姆语、泰卢固语）。每个样本包含大语言模型生成的文本、Token序列(Token)、对数概率(logits)以及源科学出版物的元数据。该数据集提供两类二元标签：
- **事实性错误**：文本是否包含幻觉内容或与事实不符的信息。
- **流畅性错误**：文本是否存在影响可读性的语言错误。

该任务将幻觉检测建模为二元分类问题，要求大语言模型对事实性与流畅性错误进行预测。

## 数据集结构
数据集分为以下划分：
| 划分 | 样本数量 | 描述 |
|-------|---------|------------|
| `train` | 1,755 | 训练集批次1（涵盖英语、印地语、西班牙语、法语、意大利语） |
| `validation` | 1,200 | 验证集（涵盖英语、印地语、西班牙语、法语、意大利语） |
| `test` | 4,384 | 测试集（涵盖全部9种语言，包含印度语言孟加拉语(bn)、泰卢固语(te)、马拉雅拉姆语(ml)、古吉拉特语(gu)）；为防止数据泄露，未提供标签。如需获取更多信息，请联系论文作者。

每个样本包含以下字段：
- `index`：唯一标识符
- `title`、`abstract`、`doi`、`url`、`datafile`：源出版物元数据
- `authors`：作者姓名列表（包含`first`名与`last`姓）
- `question`：关于该出版物的问题
- `model_id`：用于生成文本的大语言模型
- `model_config`：模型配置参数
- `prompt`：生成时使用的提示词
- `output_text`：大语言模型生成的回答
- `output_tokens`：模型输出的Token序列(Token)
- `output_logits`：Token级对数概率(logits)
- `has_fluency_mistakes`：二元标签（`y`/`n`），测试集此字段为`null`
- `has_factual_mistakes`：二元标签（`y`/`n`），测试集此字段为`null`

## 数据集来源
- Sinha, Aman 等 (2025)。[SHROOM-CAP：跨语言出版物分析中的幻觉与相关可观测生成过度错误共享任务](https://aclanthology.org/2025.chomps-main.7/)。收录于《CHOMPS 2025会议论文集》。

## 引用格式
bibtex
@inproceedings{sinha-etal-2025-shroom,
title = "{SHROOM}-{CAP}: Shared Task on Hallucinations and Related Observable Overgeneration Mistakes in Crosslingual Analyses of Publications",
author = "Sinha, Aman and
Gamba, Federica and
Vázquez, Raúl and
Mickus, Timothee and
Chattopadhyay, Ahana and
Zanella, Laura and
Arakkal Remesh, Binesh and
Kankanampati, Yash and
Chandramania, Aryan and
Agarwal, Rohit",
editor = {Sinha, Aman and
Vázquez, Raúl and
Mickus, Timothee and
Agarwal, Rohit and
Buhnila, Ioana and
Schmidtová, Patrícia and
Gamba, Federica and
Prasad, Dilip K. and
Tiedemann, Jörg},
booktitle = "Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025)",
month = dec,
year = "2025",
address = "Mumbai, India",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.chomps-main.7/",
pages = "70--80",
ISBN = "979-8-89176-308-1",
}