Research map / Large language models / Language, safety and interpretability
MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMAR, a 1,000-item multiple-choice benchmark of real-world internet audio (sound, music, speech, and mixes) with CoT rationales, and evaluates 30 audio-capable models to show that deep audio reasoning remains hard.
Problem
Existing audio-language benchmarks focus on surface-level understanding within single audio domains and lack a rigorous deep-reasoning benchmark for audio, especially for mixed-modality audio.
Approach
A hierarchical four-layer (Signal, Perception, Semantic, Cultural) taxonomy built via Human-LLM collaboration, with expert-authored, newly collected in-the-wild audio of single and mixed modalities, multi-stage human quality inspection, and CoT annotation for each question.
Key results
- Qwen-2.5-Omni (7B) is the best open-source model at 56.7% average accuracy, while no open-source LALM is significantly above random guessing (29.3%).
- Gemini 2.0 Flash achieves the top average of 65.6%, above all cascaded caption+LLM/LRM setups (e.g., Caption + OpenAI o3 at 54.7%).
- Reasoning-enhanced models outperform non-reasoning counterparts, e.g., Audio-Reasoner (36.8%) over Qwen2-Audio-Instruct (30.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use explicit reasoning in Audio-Reasoner instead of non-reasoning Qwen2-Audio | Audio models on MMAR · MMAR | accuracy (%): 30.40 -> 36.80 | Helped |
| Use explicit reasoning in Audio-Reasoner instead of Qwen2-Audio-Instruct | Audio models on MMAR · MMAR | accuracy (%): 30.00 -> 36.80 | Helped |
| Use DeepSeek-R1 instead of DeepSeek-V3 for caption-based reasoning | Caption-based models · MMAR | accuracy (%): 47.60 -> 55.50 | Helped |
| Replace GPT-4o with OpenAI o1 as the caption reasoner (ablation) | Caption + Qwen2-Audio-Instruct · MMAR | accuracy (%): 50.70 -> 53.00 | Helped |
| Replace Qwen2-Audio-Instruct with Qwen-2.5-Omni as the captioning front-end (ablation) | Caption + GPT-4o · MMAR | accuracy (%): 50.70 -> 51.80 | Helped |
| Replace Qwen2-Audio-Instruct with Qwen-2.5-Omni as the captioning front-end (ablation) | Caption + OpenAI o1 · MMAR | accuracy (%): 53.00 -> 54.40 | Helped |
| Replace GPT-4o with OpenAI o1 as the caption reasoner (ablation) | Caption + Qwen-2.5-Omni · MMAR | accuracy (%): 51.80 -> 54.40 | Helped |
| Replace original audio with noise (ablation) | Qwen2-Audio-Instruct on MMAR · MMAR | accuracy | Hurt |
| Replace original audio with noise (ablation) | Qwen-2.5-Omni on MMAR · MMAR | accuracy | Hurt |
Surprising findings
- Qwen-2.5-Omni still performs slightly above random guessing when audio is replaced with noise, indicating residual language prior bias.
- Models score highest on the semantic layer and lowest on the signal layer, even though random guessing is highest on the signal layer.
Limitations
- Audio clips are limited to about 30 seconds, so long-audio reasoning is not covered.
- Multiple-choice format with regex/string-matching evaluation; language priors are still exploitable as shown by noise-input results.
- Some baselines (e.g., DeepSeek-O1/O3 descriptions) are ambiguous and many models are evaluated with captions rather than native audio.
Experimental setup
Datasets: MMAU AudioSet
Benchmarks: MMAR MMAU
Metrics: accuracy p-value under random guessing (Poisson binomial)
Baselines: Qwen2-Audio-Instruct Qwen-2.5-Omni Gemini 2.0 Flash GPT-4o Audio Audio-Reasoner DeepSeek-R1 OpenAI o1 OpenAI o3 SALMONN GAMA
Builds on: MMAU MMLU-Pro Audio-Reasoner Qwen2-Audio
Closest papers by meaning
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.94)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.92)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.92)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.91)
- Incentivizing Consistent, Effective and Scalable Reasoning Capability in Audio LLMs via Reasoning Process Rewards (ICLR 2026 · similarity 0.91)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.