Research map / Large language models / Language, safety and interpretability
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMAU, a 10k-item multiple-choice benchmark across speech, sound, and music testing 27 expert-level information-extraction and reasoning skills, and evaluates 18 audio-language and text-only models, finding the best at about 53% vs. 82% human.
Problem
Existing audio benchmarks mostly test basic perception (ASR, acoustic scene classification) or narrow reasoning, leaving no comprehensive evaluation of expert-level audio perception and reasoning in large audio-language models.
Approach
A expert-annotated, multiple-choice benchmark spanning speech, sound, and music with 27 skills split into information extraction and reasoning, with GPT-4-augmented distractor options, plus analyses of caption-based cascades, noise-replacement tests, and error types.
Key results
- Gemini Pro achieves 52.97% average test accuracy on MMAU, the best LALM; Qwen2-Audio-Instruct reaches 52.50%.
- Human evaluators reach 82.23% average on test-mini, far above the best models.
- Cascading strong Qwen2-Audio captions into GPT-4o yields 58.74% average test accuracy, the best overall result.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide strong rather than weak audio captions to GPT-4o | GPT-4o on MMAU · MMAU test—Avg | accuracy: 45.70 -> 57.30 | Helped |
| Provide strong rather than weak audio captions to Llama-3-Instruct | Llama-3-Instruct on MMAU · MMAU test—Avg | accuracy: 42.10 -> 52.10 | Helped |
| Use audio captions before prompting text-only LLMs | Text-only LLMs on MMAU · MMAU test | accuracy | Helped |
| Replace weak captions with strong, detailed captions | Text-only LLMs on MMAU · MMAU test | accuracy | Helped |
| Replace original audio with Gaussian noise | MuLLaMa and SALMONN on MMAU · MMAU test | accuracy | No clear effect |
| Replace original audio with Gaussian noise | GAMA, Qwen2-Audio-Instruct, and Gemini Pro on MMAU · MMAU test | accuracy | Hurt |
| Use Qwen2-Audio-Instruct instead of Qwen2-Audio | Qwen2-Audio on MMAU · MMAU test—Avg | accuracy: 06.20 -> 52.50 | Helped |
| Increase question difficulty from medium to hard | Models evaluated on MMAU · MMAU | accuracy: 43.82; 36.03 | Mixed |
Surprising findings
- Text-only GPT-4o with strong audio captions outperforms the audio-native models Gemini Pro and Qwen2-Audio-Instruct.
- Some models (MuLLaMa, SALMONN) show little accuracy change when audio is replaced with Gaussian noise, suggesting reliance on language priors.
Limitations
- Only multiple-choice format; no open-ended generation evaluated.
- Information extraction and reasoning skills are kept disjoint.
- Risk of bias from human and GPT-4-driven annotation and option augmentation.
Experimental setup
Datasets: AudioSet AudioSet Strong MUStARD MELD VoxCeleb IEMOCAP MusicCaps MusicBench MTG-Jamendo SDD GuitarSet MUSDB18
Benchmarks: MMAU
Metrics: accuracy
Baselines: Qwen2-Audio-Instruct Gemini Pro GAMA SALMONN LTU-AS Audio Flamingo Chat GPT-4o Llama-3-8B-Instruct
Builds on: AudioSet MusicCaps MELD IEMOCAP
Closest papers by meaning
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.94)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.93)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.92)
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models (NeurIPS 2025 · similarity 0.90)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.89)
- STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.