Helsinki-NLP/shroom

SHROOM数据集是一个用于检测自然语言生成系统中的虚构和过度生成错误的数据集。它包含4000个模型输出的样本，每个样本由5个标注者进行标注，覆盖了机器翻译、释义生成和定义建模三个NLP任务。该数据集用于共享任务，共有58名用户组成的42个团队参与，其中26个团队撰写了系统描述论文。参与者提交了超过300个预测集。数据集的结构包括验证集和测试集，分为不依赖特定模型的model-agnostic配置和依赖特定模型的model-aware配置。

The SHROOM dataset is designed for detecting hallucinations and overgeneration mistakes in natural language generation systems. It consists of 4000 model output samples, each annotated by 5 annotators, spanning across three NLP tasks: machine translation, paraphrase generation, and definition modeling. The dataset is used for a shared task, with participation from 58 users across 42 teams, 26 of which wrote system description papers. Participants submitted over 300 prediction sets. The dataset structure includes validation and test splits, with configurations for model-agnostic and model-aware approaches.