ATH-MaaS/HalloMTBench

--- license: apache-2.0 task_categories: - translation --- # HalloMTBench: A Benchmark for Translation Hallucination in LLMs **[ Paper ](https://arxiv.org/abs/2510.24073) | [ GitHub ](https://github.com/AIDC-AI)** --- ## Dataset Summary **HalloMTBench** is a new and challenging benchmark designed to evaluate the performance of Large Language Models (LLMs) against translation hallucinations. The result is a high-quality, expert-verified dataset of **6,908 challenging samples** that capture naturally occurring hallucinations, providing a cost-effective and robust tool for evaluating model safety and reliability in translation tasks. ## Supported Tasks and Leaderboards The primary use of this dataset is for **evaluating the robustness of LLMs against translation hallucinations**. Models can be prompted to translate the `source_text` and their output can be compared against the `target_text` and `halluc_type` to measure their susceptibility to hallucination. An official leaderboard and evaluation tool, **HalloMTDetector**, are available in the [repository](https://github.com/AIDC-AI/). ## Languages The dataset covers **11 high-resource language pairs**, with English (`en`) as the source language. * **Source Language:** English (`en`) * **Target Languages:** Spanish (`es`), French (`fr`), Italian (`it`), Portuguese (`pt`), German (`de`), Russian (`ru`), Arabic (`ar`), Vietnamese (`vi`), Chinese (`zh`), Japanese (`ja`), Korean (`ko`). ## Dataset Structure ### Data Distribution The 6,908 samples in the dataset are distributed across the four hallucination types as follows. Avg. Target Length refers to the average character length of the target_text. | Hallucination Type | Count | Avg. Target Length | |-----------------------------|-------|--------------------| | Extraneous Addition | 3,688 | 135.3 | | Incorrect Language | 2,663 | 440.0 | | Untranslated Content | 501 | 6.9 | | Repetition | 56 | 114.8 | | **Total** | **6,908** | **148.7** | ### Data Instances Each instance in the dataset is a JSON object representing a single, expert-verified example of a translation hallucination. ```json { "source_text":"What will the temperature be next Tuesday?", "target_text":"ما将是下周二的温度？", "lang_pair":"en-ar", "model":"qwen-max", "halluc_type":"Incorrect Language" } ``` ## License / 许可证 The dataset is licensed under the [apache-2.0](https://www.apache.org/licenses/LICENSE-2.0). ``` @article{wu2025challenging, title={Challenging Multilingual LLMs: A New Taxonomy and Benchmark for Unraveling Hallucination in Translation}, author={Wu, Xinwei and Liu, Heng and Wang, Hao and Liu, Yangyang and Zhao, Xiaohu and Zhou, Jiang and Zeng, Bo and Dong, Tianyu and Shi, Dan and Xu, Linlong and Wang, Longyue and Xiong, Deyi and Luo, Weihua and Zhang, Kaifu}, journal={arXiv preprint arXiv:2510.24073}, year={2025} } ```--- 许可证: apache-2.0 任务类别: - 翻译 --- # HalloMTBench：大语言模型翻译幻觉评测基准 **[ 论文 ](https://arxiv.org/abs/2510.24073) | [ GitHub仓库 ](https://github.com/AIDC-AI)** --- ## 数据集概述 **HalloMTBench** 是一款全新且极具挑战性的基准测试集，旨在评测大语言模型（Large Language Models, LLMs）应对翻译幻觉的性能。 本数据集为高质量、经专家验证的数据集，包含6908条高难度样本，涵盖自然出现的翻译幻觉场景，是一款兼具成本效益与鲁棒性的工具，可用于评估翻译任务中模型的安全性与可靠性。 ## 支持任务与评测榜单 本数据集的核心用途为**评测大语言模型抵御翻译幻觉的鲁棒性**。可通过提示模型翻译`source_text`（源文本），将模型输出与`target_text`（目标文本）及`halluc_type`（幻觉类型）进行比对，以衡量模型出现幻觉的概率。 官方评测榜单与评测工具**HalloMTDetector**已在[仓库](https://github.com/AIDC-AI/)中开源。 ## 语言覆盖 本数据集覆盖**11种高资源语言对**，以英语（`en`）作为源语言。 * **源语言：** 英语（`en`） * **目标语言：** 西班牙语（`es`）、法语（`fr`）、意大利语（`it`）、葡萄牙语（`pt`）、德语（`de`）、俄语（`ru`）、阿拉伯语（`ar`）、越南语（`vi`）、中文（`zh`）、日语（`ja`）、韩语（`ko`）。 ## 数据集结构 ### 数据分布 本数据集的6908条样本按4种幻觉类型分布如下。平均目标文本长度指目标文本的平均字符长度。 | 幻觉类型 | 样本数量 | 平均目标文本长度 | |---------------------------|---------|------------------| | 额外添加内容 | 3688 | 135.3 | | 语言错误 | 2663 | 440.0 | | 未翻译内容 | 501 | 6.9 | | 重复内容 | 56 | 114.8 | | **总计** | **6908** | **148.7** | ### 数据实例 数据集中的每个实例均为JSON对象，代表一条经专家验证的翻译幻觉示例。 json { "source_text":"What will the temperature be next Tuesday?", "target_text":"ما将是下周二的温度？", "lang_pair":"en-ar", "model":"qwen-max", "halluc_type":"Incorrect Language" } ## 许可证 本数据集采用[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)开源许可证。 @article{wu2025challenging, title={Challenging Multilingual LLMs: A New Taxonomy and Benchmark for Unraveling Hallucination in Translation}, author={Wu, Xinwei and Liu, Heng and Wang, Hao and Liu, Yangyang and Zhao, Xiaohu and Zhou, Jiang and Zeng, Bo and Dong, Tianyu and Shi, Dan and Xu, Linlong and Wang, Longyue and Xiong, Deyi and Luo, Weihua and Zhang, Kaifu}, journal={arXiv preprint arXiv:2510.24073}, year={2025} }