kyutai/HaluEvalAudio_1000

HaluEvalAudio 1000 Dataset

<p align="center"> <img src="haluevalaudio_1000.png" width="500" alt="HeluEvalAUdio 1000 Logo"> </p>

Dataset Description

HaluEvalAudio 1000 is a specialized speech-based question-answering dataset designed to benchmark the capabilities of general multimodal & audio-focused language models as well as retrieval-augmented audio language models. Compared to common QA benchmarks such as Llama Questions, Web Questions, or TriviaQA, HaluEvalAudio 1000 introduces more challenging questions and topics and is specifically structured for Retrieval-Augmented Generation (RAG) evaluation.

Deriveds from the HaluEval dataset, A key feature of the dataset is that it provides ground-truth references in text format. This enables two complementary evaluation setups for RAG models: (1) End-to-End RAG, utilizing the model’s internal retrieval pipeline, and (2) Oracle-Aided Generation, where providing ground-truth context as an ablation study allows researchers to isolate retrieval quality from downstream generative performance.

Dataset Summary

• —Source: The qa subset of the HaluEval dataset.
• —Total instances: 1,000 WAV audio files synthesized with Kyutai's TTS model, with paired reference (knowledge) and ground-truth answers provided in textual format.

Data Format

Each entry in the dataset contains the following fields:

• —audio: The synthesized WAV file containing the spoken question.
• —text: The text transcription of the audio question.
• —knowledge: The ground-truth textual knowledge from the original HaluEval dataset.
• —answer: The ground-truth textual answer from the original HaluEval dataset.

Dataset Construction

We extract the first 1,000 instances from the qa subset of the HaluEval dataset. We use Kyutai's multistream TTS model (the same model is used for generating MoshiRAG's training data, but with voices sampled from a different dataset), and randomly sample speaker voices from the Common Voice dataset to convert the textual questions into audio. The original textual knowledge and ground-truth answers from HaluEval are preserved, while the hallucinated_answer field is removed for simplicity.

Citations

If you use this dataset, please cite:

@misc{chien2026moshirag,
      title={MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models}, 
      author={Chung-Ming Chien and Manu Orsini and Eugene Kharitonov and Neil Zeghidour and Karen Livescu and Alexandre D{\'e}fossez},
      year={2026},
      eprint={2604.12928},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.12928}, 
}

Acknowledgment & Licensing

This dataset is a derivative work. Audio Files are licensed under CC BY-NC 4.0. Source Text is derived from the Halueval dataset which is licensed under the MIT License (Copyright (c) 2020 RUCAIBox).