Harland/DCASE2026-Task5-DevSet
DCASE 2026 Task 5 Audio-Dependent Question Answering (ADQA) Development Set This is the official Development Set for DCASE 2026 Challenge Task 5: Audio-Dependent Question Answering (ADQA). The ADQA task focuses on addressing "Textual Hallucination" in Large Audio-Language Models (LALMs) — where models pass audio understanding benchmarks by relying on text prompts and internal linguistic priors rather than actual audio perception. ADQA introduces a rigorous evaluation… See the full description on the dataset page: https://huggingface.co/datasets/Harland/DCASE2026-Task5-DevSet.
1563
1---2configs:3- config_name: default4 data_files:5 - split: dev6 path: "dev.jsonl"7license: apache-2.08---9# DCASE 2026 Task 5 Audio-Dependent Question Answering (ADQA) Development Set10 11<div align="center">12 13[](https://dcase.community/challenge2026/task-audio-dependent-question-answering)14[](https://arxiv.org/abs/2509.21060)15[](https://huggingface.co/datasets/Harland/AudioMCQ-StrongAC-GeminiCoT)16 17</div>18 19This is the official **Development Set** for [DCASE 2026 Challenge Task 5: Audio-Dependent Question Answering (ADQA)](https://dcase.community/challenge2026/task-audio-dependent-question-answering).20 21The ADQA task focuses on addressing **"Textual Hallucination"** in Large Audio-Language Models (LALMs) — where models pass audio understanding benchmarks by relying on text prompts and internal linguistic priors rather than actual audio perception. ADQA introduces a rigorous evaluation framework using **Audio-Dependency Filtering (ADF)** to ensure questions cannot be answered through common sense or text-only reasoning.22 23## Audio-Dependency Filtering (ADF)24 25All samples in this development set undergo a rigorous four-step ADF hard-filtering process to guarantee genuine audio dependence:26 271. **Silent Audio Filtering:** Questions solvable by LALMs without audio are removed.282. **LLM Common-sense Check:** Ensures no external knowledge alone can solve the question.293. **Perplexity-based Soft Filtering:** Eliminates samples with text-based statistical shortcuts.304. **Manual Verification:** Final human-in-the-loop check for ground-truth accuracy.31 32## Statistics33 34| Metric | Count |35|--------|-------|36| Total Samples | 1,607 |37| Unique Audio Files | 1,607 |38 39### Data Sources40 41The development set is composed of two parts:42 43- **Existing Benchmarks:** A portion of the samples is derived from established audio understanding benchmarks, including [MMAU](https://github.com/sakshi113/mmau), [MMAR](https://github.com/ddlBoJack/MMAR), and [MMSU](https://huggingface.co/datasets/ddwang2000/MMSU). These samples cover a wide range of audio understanding tasks such as speech, music, and sound perception.44- **Human-Annotated Questions:** The remaining majority consists of newly constructed, human-annotated multiple-choice questions based on diverse audio sources, designed to further challenge models on real-world audio comprehension.45 46All samples undergo the four-step **Audio-Dependency Filtering (ADF)** process described above.47 48## Directory Structure49 50```text51DCASE2026-Task5-DevSet/52├── dev.jsonl # Main data file (1,607 samples, shuffled)53├── dev_audios/ # Audio files (1,607 .wav files)54└── README.md55```56 57## Data Format58 59Each entry in `dev.jsonl` is a JSON object with the following fields:60 61| Field | Type | Description |62|-------|------|-------------|63| `id` | string | Unique sample identifier (e.g., `dev_0001`) |64| `audio_path` | string | Relative path to audio file |65| `question_text` | string | Question text |66| `answer` | string | Correct answer |67| `multi_choice` | list[string] | Answer choices |68 69### Example70 71```json72{73 "id": "dev_0001",74 "audio_path": "dev_audios/dev_0001.wav",75 "question_text": "What is the speaker's primary emotion in this audio?",76 "answer": "Happiness",77 "multi_choice": ["Sadness", "Happiness", "Anger", "Fear"]78}79```80 81## Submission Format82 83The system output file should be a `.csv` file with the following two columns:84 85| Column | Description |86|--------|-------------|87| `question` | The question ID (e.g., `dev_0001`) |88| `answer` | The system's answer, must match one of the given choices |89 90## License91 92This dataset is distributed under the **Apache-2.0** license.93 94## Citation95 96If you use this development set or participate in DCASE 2026 Task 5, please cite:97 98```bibtex99@article{he2025measuring,100 title={Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models},101 author={He, Haolin and Du, Xingjian and Sun, Renhe and Dai, Zheqi and Xiao, Yujia and Yang, Mingru and Zhou, Jiayi and Li, Xiquan and Liu, Zhengxi and Liang, Zining and others},102 journal={arXiv preprint arXiv:2509.21060},103 year={2025}104}105 106@article{he2026summary,107 title={Summary of DCASE 2026 Task 5: Audio-Dependent Question Answering},108 author={He, Haolin and Sun, Renhe and Dai, Zheqi and Du, Xingjian and Wu, Chunyat and Liang, Zining and Liu, Zhengxi and Lei, Jiahe and Wang, Runbang and Zhou, Jiayi and Yang, Mingru and Li, Xiquan and Chen, Yun and Chen, Xie and Duan, Zhiyao and Wang, Weiqiang and Plumbley, Mark D. and Liu, Jian and Kong, Qiuqiang},109 journal={arXiv preprint arXiv:2607.18718},110 year={2026}111}112```113