HaluEvalAudio_1000

# HaluEvalAudio 1000 Dataset <p align="center"> <img src="haluevalaudio_1000.png" width="500" alt="HeluEvalAUdio 1000 Logo"> </p> ## Dataset Description **HaluEvalAudio 1000** is a specialized speech-based question-answering dataset designed to benchmark the capabilities of **general multimodal & audio-focused language models** as well as **retrieval-augmented audio language models**. Compared to common QA benchmarks such as Llama Questions, Web Questions, or TriviaQA, HaluEvalAudio 1000 introduces more challenging questions and topics and is specifically structured for Retrieval-Augmented Generation **(RAG) evaluation**. Deriveds from the [HaluEval dataset](https://aclanthology.org/2023.emnlp-main.397/), A key feature of the dataset is that it provides **ground-truth references** in text format. This enables two complementary evaluation setups for RAG models: (1) End-to-End RAG, utilizing the model’s internal retrieval pipeline, and (2) Oracle-Aided Generation, where providing ground-truth context as an ablation study allows researchers to isolate retrieval quality from downstream generative performance. --- ## Dataset Summary * **Source:** The `qa` subset of the HaluEval dataset. * **Total instances:** 1,000 WAV audio files synthesized with Kyutai's TTS model, with paired reference (knowledge) and ground-truth answers provided in textual format. --- ## Data Format Each entry in the dataset contains the following fields: - `audio`: The synthesized WAV file containing the spoken question. - `text`: The text transcription of the `audio` question. - `knowledge`: The ground-truth textual knowledge from the original HaluEval dataset. - `answer`: The ground-truth textual answer from the original HaluEval dataset. --- ## Dataset Construction We extract the first 1,000 instances from the `qa` subset of the HaluEval dataset. We use Kyutai's multistream TTS model (the same model is used for generating MoshiRAG's training data, but with voices sampled from a different dataset), and randomly sample speaker voices from the [Common Voice](https://www.mozillafoundation.org/en/common-voice/) dataset to convert the textual questions into audio. The original textual knowledge and ground-truth answers from HaluEval are preserved, while the `hallucinated_answer` field is removed for simplicity. --- ## Citations If you use this dataset, please cite: ```bibtex @misc{chien2026moshirag, title={MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models}, author={Chung-Ming Chien and Manu Orsini and Eugene Kharitonov and Neil Zeghidour and Karen Livescu and Alexandre D{\'e}fossez}, year={2026}, eprint={2604.12928}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2604.12928}, } ``` ## Acknowledgment & Licensing This dataset is a derivative work. Audio Files are licensed under CC BY-NC 4.0. Source Text is derived from the Halueval dataset which is licensed under the MIT License (Copyright (c) 2020 RUCAIBox).