Research map / Vision-language and multimodal / Multimodal LLMs
AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AudSemThinker, a Qwen2.5-Omni-7B audio-language model trained with SFT and GRPO to reason over auditory semantic descriptors, plus the AudSem dataset (~800k-900k examples) built from YouTube sound-description subtitles and multi-model audio/visual analysis.
Problem
Audio-language models lack explicit reasoning over fine-grained sound semantics, and existing audio caption datasets overlap heavily, causing data contamination in zero-shot evaluation.
Approach
Structure audio reasoning around a who/what/how/when/where semantic-descriptor framework in a thinking phase before answering, trained on a new low-overlap dataset built from YouTube SDH closed captions filtered by caption-subtitle consistency.
Key results
- AudSemThinker-QA reaches 65.70 average on MMAU Test (vs. Qwen2.5-Omni-7B 65.60 on Test-Mini), and AudSemThinker-QA GRPO reaches 66.03 on MMAU Test.
- AudSemThinker achieves 62.90 Test-Mini / 64.41 Test average on MMAU, above Qwen2-Audio (49.20 / 52.50) and Audio Flamingo 2 (55.48 / 59.42) under SFT.
- On AudioBench, AudSemThinker obtains the best AudioCaps QA (62.81) and WavCaps QA (55.20) scores and MuchoMusic (76.66).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on AudSem with structured semantic reasoning | Qwen2.5-Omni-7B · MMAU Test | accuracy: 52.50 -> 64.41 | Helped |
| Fine-tune on AudSem multiple-choice QA using GRPO | Qwen2.5-Omni-7B · MMAU Test-mini | accuracy: 65.60 -> 66.70 | Helped |
| Use GRPO rather than SFT for QA training | AudSemThinker-QA · MuchoMusic | score: 74.47 -> 62.00 | Hurt |
| Fine-tune on AudSem with semantic descriptors (ablation) | Qwen2.5-Omni-7B · MMAU Test-mini, QA subset | accuracy: 61.90 -> 63.90 | Helped |
| Fine-tune on AudSem with semantic descriptors (ablation) | Qwen2-Audio-7B-Instruct · MMAU Test-mini, QA subset | accuracy: 59.50 -> 61.50 | Helped |
| Remove LoRA from SFT (ablation) | Qwen2.5-Omni-7B · MMAU Test-mini, QA subset | accuracy: 63.90 -> 15.50 | Hurt |
| Remove LoRA from SFT (ablation) | Qwen2-Audio-7B-Instruct · MMAU Test-mini, QA subset | accuracy: 61.50 -> 23.60 | Hurt |
| Use full AudSem training data instead of QA-only data (ablation) | Qwen2.5-Omni-7B with semantic descriptors and LoRA · MMAU Test-mini | accuracy: 63.90 -> 62.90 | Hurt |
| Remove semantic descriptors from full-data SFT (ablation) | Qwen2.5-Omni-7B · MMAU Test-mini | accuracy: 62.90 -> 60.50 | Hurt |
| Remove semantic descriptors from GRPO (ablation) | Qwen2.5-Omni-7B · MMAU Test-mini | accuracy: 61.00 -> 62.00 | Helped |
| Remove semantic descriptors from GRPO (ablation) | Qwen2-Audio-7B-Instruct · MMAU Test-mini | accuracy: 57.10 -> 56.80 | Hurt |
| Remove LoRA from GRPO (ablation) | Qwen2.5-Omni-7B · MMAU Test-mini | accuracy: 62.00 -> 49.70 | Hurt |
Surprising findings
- Semantic descriptors did not consistently help in the GRPO setting, possibly because the reward targets only final answer correctness.
- RLVR-trained model performs well on closed-caption benchmarks but fails to generalize to AudioBench open-ended QA.
Limitations
- Speech understanding is weaker, since no speech-to-text models were used in the data pipeline.
- Dataset captions are generated by a multi-model pipeline with noisy intermediate outputs and a moderate annotator agreement (Krippendorff's Alpha 0.538).
- Possible contamination: the pretrained Qwen2.5-Omni may have seen test data, and MMAU filename obfuscation limits certainty about training overlap.
Experimental setup
Datasets: AudSem AudioSet AudioCaps VGGSound LP-MusicCaps
Benchmarks: MMAU AudioBench
Metrics: accuracy model-judged score
Baselines: Qwen2-Audio Qwen2.5-Omni-7B Audio Flamingo 2 Gemini 2.0 Flash Kimi-Audio Mellow R1-AQA Audio-Reasoner Audio-CoT
Builds on: Qwen2.5-Omni GRPO LCPO Audio-Reasoner LoRA
Closest papers by meaning
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.90)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.89)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.89)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.89)
- Audio Entailment: Assessing Deductive Reasoning for Audio Understanding (AAAI 2025 · similarity 0.89)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.