Research map / Large language models / Language, safety and interpretability
Mellow: a small audio language model for reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Mellow, a 167M-parameter small audio-language model for reasoning, trained on the ReasonAQA dataset (70% LLM-generated QA from AudioCaps/Clotho captions), and benchmarks it on MMAU, audio entailment, audio difference, and captioning/AQA.
Problem
Audio-language models that can reason over audio and text typically exceed 8B parameters; no prior work has explored small (sub-billion) audio-language models for audio reasoning, which matters for edge devices.
Approach
Build a small ALM (HTSAT audio encoder, non-linear projection, SmolLM2 LM, full finetuning) and train it on a controlled, audio-restricted reasoning dataset (ReasonAQA) of LLM-generated detailed and MCQ questions grounded in AudioCaps and Clotho captions, so gains come from design and data style rather than audio scale.
Key results
- Mellow (167M) scores 52.11 test / 48.40 test-mini average on MMAU, close to Qwen2-Audio-Instruct (52.50 test) at 8.4B parameters, and is the best small audio-language model on it.
- On audio entailment CLE, Mellow reaches 91.16 accuracy versus 52.22 for SALMONN (13B) in zero-shot evaluation.
- On ClothoAQA binary QA, Mellow achieves 71.4% accuracy versus Pengi's 63.6%, and captioning SPICE 17.8 versus Pengi's 12.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the linear projection with a transformer projection while GPT2 is frozen (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 26.05 -> 30.19 | Helped |
| Replace the non-linear projection with a transformer projection while GPT2 is fine-tuned (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 41.29 -> 41.36 | No clear effect |
| Replace GPT2 with SmolLM2 while keeping the LM frozen (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 25.89 -> 28.60 | Helped |
| Fine-tune SmolLM2 instead of freezing it (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 28.60 -> 46.29 | Helped |
| Use full fine-tuning instead of prefix-tuning (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 28.60 -> 46.29 | Helped |
| Use full fine-tuning instead of LoRA (8, 16) (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 39.10 -> 46.29 | Helped |
| Use full fine-tuning instead of LoRA (256, 512) (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 44.40 -> 46.29 | Helped |
| Use HTSAT instead of CNN14 as the audio encoder (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 43.30 -> 46.29 | Helped |
| Replace Type 1 data with Type 2 synthetic data (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 46.29 -> 46.70 | No clear effect |
| Replace Type 1 data with Type 3 synthetic data (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 46.29 -> 46.40 | No clear effect |
| Combine Type 1, Type 2, and Type 3 synthetic data as Type 4 (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 46.29 -> 48.40 | Helped |
| Add WavCaps training data (ablation) | Mellow · MMAU Test-mini | Avg. accuracy: 46.29 -> 47.80 | Helped |
Surprising findings
- Mellow performs near random on speech-based MMAU tasks and shows only about a 13% drop when audio is replaced by Gaussian noise, indicating limited audio grounding.
- Replacing GPT-2 with SmolLM2 improves open-ended reasoning, but deductive reasoning performance is comparable unless a larger LM (over 1B parameters) is used.
Limitations
- Performs poorly on speech-related tasks because it is not trained on speech data.
- Weaker at detecting plausible (neutral) hypotheses in audio entailment, with about 75% accuracy on NACC for ACE.
- Lags larger LALMs on captioning and on Tier-2 audio difference BLEU-4 due to a smaller language model.
Experimental setup
Datasets: AudioCaps Clotho ClothoAQA CLE ACE CLD ACD MMAU
Benchmarks: MMAU CLE ACE CLD ACD ClothoAQA AudioCaps captioning
Metrics: accuracy SPICE BLEU-4 precision recall F1
Baselines: Pengi Audio Flamingo Chat Qwen-Audio-Chat Qwen2-Audio-Instruct GAMA ADIFF CLAP SALMONN
Builds on: Pengi OpenAQA HTSAT SmolLM2
Closest papers by meaning
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.92)
- Audio Entailment: Assessing Deductive Reasoning for Audio Understanding (AAAI 2025 · similarity 0.92)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.91)
- UALM: Unified Audio Language Model for Understanding, Generation and Reasoning (ICLR 2026 · similarity 0.91)
- AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound (NeurIPS 2025 · similarity 0.90)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.