Research map / Vision-language and multimodal / Multimodal LLMs

MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMSU, a 5,000-question, 47-task multiple-choice benchmark grounded in linguistic theory for evaluating spoken language understanding and reasoning in SpeechLLMs, and evaluates 22 models against human performance.

Problem

Existing speech benchmarks for SpeechLLMs focus on semantic content, rely heavily on TTS audio, and ignore linguistic phenomena such as prosody, phonetics, rhetoric and paralinguistics, leaving fine-grained speech perception and reasoning under-evaluated.

Approach

Organize speech understanding tasks with a linguistics-based hierarchy (perception vs. reasoning, linguistics vs. paralinguistics) and build mostly real (open-source and custom-recorded) audio MCQs with expert-in-the-loop option augmentation via GPT-4o, then benchmark a broad set of SpeechLLMs and OmniLLMs.

Key results

• Gemini-1.5-Pro achieves the best average accuracy of 60.68% on MMSU, versus 89.72% for human evaluators.
• Open-source Qwen2.5-Omni-7B reaches 60.57% average accuracy, nearly matching Gemini-1.5-Pro and above GPT-4o-Audio (56.38%).
• Phonology perception is a bottleneck: the best phonology perception score is 53.60% (Gemini-1.5-Pro), with many models near chance on near-homophone and consonant-vowel perception.

Surprising findings

• GPT-4o-Audio underperforms many open-source models, with the paper attributing this partly to refusals on speaker-trait questions such as gender prediction.
• Models perform worse on perception than on reasoning tasks, while humans show the opposite pattern (91.24% perception vs. 86.77% reasoning).

Limitations

• Benchmark is multiple-choice only (four options), which may not capture open-ended speech understanding.
• A portion of audio (9.82%) is synthetic Azure TTS, and some tasks use small numbers of custom recordings.
• Error analysis is based on a random sample of 300 mispredictions per model.

Experimental setup

Datasets: MELD GigaSpeech CommonVoice Emilia CoVoST 2 EDACC VCTK CHILDES SLURP SEAME Fake-or-Real RAVDESS

Benchmarks: MMSU

Metrics: accuracy

Baselines: Random Guess Most Frequent Choice Human Kimi-Audio Qwen2.5-Omni-7B Gemini-1.5-Pro GPT-4o-Audio Qwen2-Audio-Instruct

Builds on: MMAU AIR-Bench Dynamic-SUPERB VoiceBench

Closest papers by meaning

• MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.91)
• Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.90)
• MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.89)
• Listen, Think, and Understand (ICLR 2024 · similarity 0.89)
• Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.89)
• MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks (ICML 2026 · similarity 0.89)