ddwang2000/MMSU
[ICLR 2026] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark Overview of MMSU MMSU (Massive Multi-task Spoken Language Understanding and Reasoning Benchmark) is a comprehensive benchmark for evaluating fine-grained spoken language understanding and reasoning in multimodal models. It systematically captures the variance of real-world linguistic phenomena in daily speech through 47 sub-tasks, including phonetics, prosody, rhetoric… See the full description on the dataset page: https://huggingface.co/datasets/ddwang2000/MMSU.
[ICLR 2026] MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
 
Overview of MMSU
MMSU (Massive Multi-task Spoken Language Understanding and Reasoning Benchmark) is a comprehensive benchmark for evaluating fine-grained spoken language understanding and reasoning in multimodal models.
It systematically captures the variance of real-world linguistic phenomena in daily speech through 47 sub-tasks, including phonetics, prosody, rhetoric, syntactics, semantics, and paralinguistics, spanning both perceptual and higher-level reasoning capabilities.
The benchmark comprises 5,000 carefully curated audio–question–answer pairs derived from diverse authentic recordings.
Usage
You can load the dataset via Hugging Face datasets:
from datasets import load_dataset
ds = load_dataset("ddwang2000/MMSU")For evaluation, please refer to **GitHub Code**
Citation
@article{wang2025mmsu,
title={MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark},
author={Dingdong Wang and Jincenzi Wu and Junan Li and Dongchao Yang and Xueyuan Chen and Tianhua Zhang and Helen Meng},
journal={arXiv preprint arXiv:2506.04779},
year={2025},
}