Research map / Large language models / Language, safety and interpretability
Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Audio Flamingo 2, a 3B-LLM audio-language model with a custom CLAP encoder (AF-CLAP), synthetic reasoning QA (AudioSkills), and 3-stage curriculum, extended to long audio (up to 5 min) via LongAudio dataset and LongAudioBench.
Problem
Audio-language models underperform on expert-level audio reasoning and are limited to about 30 seconds of audio input, with no large dataset or benchmark for long-audio understanding.
Approach
Combine a CLAP encoder retrained on 8M audio-caption pairs with linguistic-variation positives and composition-aware negatives, skill-specific synthetic QA data, gated cross-attention to a small 3B LLM, and a curriculum that extends audio context from 30 s to 5 min using a new long-audio QA dataset.
Key results
- AF2 with a 3B LLM reaches 65.1% on MMAU Sound and 72.9% on MMAU Music, versus previous best 61.7% and 56.5% (Gemini Flash v2).
- On LongAudioBench, AF2 scores 64.2% versus 45.3% for the previous best (Gemini Flash v2).
- AF-CLAP reaches 0.58 CIDEr on AudioCaps and 69.1 on GTZAN, beating Laion-CLAP (0.51 and 65.2) in the same setup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train AF2 on LongAudio (ablation) | Audio Flamingo 2 · LongAudioBench | judge score: 50.22% -> 64.19% | Helped |
| Use dense audio encoder features instead of CLAP-head features (ablation) | Audio Flamingo 2 · MMAU | score: 47.0 -> 69.0 | Helped |
| Add AudioSkills to OpenAQA training data (ablation) | Audio Flamingo 2 · MMAU (avg) | score: 38.2 -> 57.5 | Helped |
| Add AudioSkills to OpenAQA training data (ablation) | Audio Flamingo 2 · MuchoMusic | score: 39.8 -> 51.3 | Helped |
| Use the full training data instead of half (ablation) | Audio Flamingo 2 · MMAU (avg) | score: 49.3 -> 69.0 | Helped |
| Use the full training data instead of one-third (ablation) | Audio Flamingo 2 · MMAU (avg) | score: 45.1 -> 69.0 | Helped |
| Use AF-CLAP instead of AF-CLAP with standard contrastive loss (ablation) | Audio Flamingo 2 · MMAU (avg) | score: 66.8 -> 69.0 | Helped |
| Use AF-CLAP instead of AF-CLAP with standard contrastive loss (ablation) | Audio Flamingo 2 · MuchoMusic | score: 54.4 -> 56.5 | Helped |
| Use cross-attention conditioning instead of prefix tuning (ablation) | Audio Flamingo 2 · MMAU (avg) | score: 52.8 -> 69.0 | Helped |
| Use cross-attention conditioning instead of prefix tuning (ablation) | Audio Flamingo 2 · MuchoMusic | score: 40.3 -> 56.5 | Helped |
| Fine-tune the LLM during training (ablation) | Audio Flamingo 2 · MMAU | score: 69.0 -> 59.1 | Hurt |
| Use a three-stage curriculum rather than one-stage training (ablation) | Audio Flamingo 2 · MMAU | score: 58.8 -> 69.0 | Helped |
Surprising findings
- Fine-tuning the LLM improves retrieval-based classification partly through style memorization, which favors CLAP-based label retrieval, while hurting reasoning performance.
- AF-CLAP trained on Laion-CLAP's 630k pairs does not improve over Laion-CLAP, so gains come from the added data and cleaning.
Limitations
- Weak speech content understanding.
- CLAP-based retrieval evaluation can mis-score open-ended generations.
- Encoder not designed for long audio; long audio is handled by concatenating 10-second window features.
Experimental setup
Datasets: AudioSkills LongAudio MiraData Video ReCap AudioSet-SL MusicBench OpenAQA YouTube-8M Sound-VECaps
Benchmarks: MMAU LongAudioBench AudioCaps GTZAN MuchoMusic CompA-R-test OpenAQA ClothoAQA MusicInstruct MusicQA CMM ESC50
Metrics: accuracy CIDEr GPT-4o judge score (1-10) retrieval recall
Baselines: Gemini-1.5-Pro Gemini Flash v2 GPT-4o-audio GAMA Qwen2-Audio Qwen-Audio Audio Flamingo SALMONN LTU Laion-CLAP MS-CLAP CompA-CLAP
Builds on: CLAP Flamingo Audio Flamingo GAMA OpenAQA
Closest papers by meaning
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models (NeurIPS 2025 · similarity 0.94)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.92)
- Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities (ICML 2024 · similarity 0.92)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.92)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.92)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.