Research map / Large language models / Language, safety and interpretability
Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Audio Flamingo 3 is a fully open large audio-language model that unifies speech, sound, and music understanding via the AF-Whisper encoder, on-demand thinking, long-audio and multi-audio chat, and voice output, trained with a five-stage curriculum on open-source data.
Problem
Existing large audio-language models are mostly trained on short recognition-style audio, lack reasoning, multi-turn/multi-audio chat, and long-audio (speech included) understanding, and most multi-modality (speech, sound, music) LALMs are closed or only partially open.
Approach
A unified audio encoder (AF-Whisper, initialized from Whisper large-v3 and trained on captioning over speech, sound, and music) feeds Qwen2.5-7B, combined with large skill-focused QA datasets (AudioSkills-XL, LongAudio-XL, AF-Think, AF-Chat), a lightweight prompt-triggered thinking mode, and a five-stage curriculum.
Key results
- AF3 reaches 72.42 average accuracy on MMAU test (vs 71.00 for Qwen2.5-Omni) and 74.26 with +Think on MMAU test-mini.
- On AF-Chat-test, AF3-Chat scores 3.6/3.4/3.9 on factuality/usefulness/depth vs 2.4/2.7/3.2 for Qwen2.5-Omni.
- AF3 achieves LibriSpeech test-clean WER 1.57 and test-other 3.13, beating Phi-4-mm and Qwen2.5-Omni.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Enable the thinking prompt | Audio Flamingo 3 · MMAU-v05.15.25 test (Avg) | accuracy: 72.42 -> 73.16 | Helped |
| Enable the thinking prompt | Audio Flamingo 3 · MMAU-v05.15.25 test-mini (Avg) | accuracy: 73.30 -> 74.26 | Helped |
| Enable the thinking prompt | Audio Flamingo 3 · MMAR | accuracy: 58.5 -> 60.1 | Helped |
| Enable the thinking prompt | Audio Flamingo 3 · MMSU | accuracy: 61.4 -> 62.3 | Helped |
| Enable the thinking prompt | Audio Flamingo 3 · MMAU-v05.15.25 test Sound | accuracy: 75.83 -> 75.27 | Hurt |
| Replace unified AF-Whisper with dual CLAP and Whisper-v3 encoders (ablation) | Audio Flamingo 3 · MMAU-Sound (10% data) | accuracy: 66.7 -> 63.7 | Hurt |
| Replace unified AF-Whisper with dual CLAP and Whisper-v3 encoders (ablation) | Audio Flamingo 3 · MMAU-Music (10% data) | accuracy: 65.9 -> 68.3 | Helped |
| Replace unified AF-Whisper with dual CLAP and Whisper-v3 encoders (ablation) | Audio Flamingo 3 · MMAU-Speech (10% data) | accuracy: 57.4 -> 45.2 | Hurt |
| Remove AudioSkills-XL from Stage 3 (ablation) | Audio Flamingo 3 · MMAU-Sound (10% data) | accuracy: 66.7 -> 56.1 | Hurt |
| Remove AudioSkills-XL from Stage 3 (ablation) | Audio Flamingo 3 · MMAU-Music (10% data) | accuracy: 65.9 -> 42.1 | Hurt |
| Remove AudioSkills-XL from Stage 3 (ablation) | Audio Flamingo 3 · MMAU-Speech (10% data) | accuracy: 57.4 -> 14.3 | Hurt |
| Use clustered rather than randomly selected audio clips for AF-Chat (ablation) | AF3-Chat · AF-Chat-test | response relevance and conversational depth | Helped |
Surprising findings
- Deep explicit thinking does not always improve audio understanding, so thinking is enabled on demand via a prompt suffix.
- AF3 reaches state-of-the-art ASR on several benchmarks without training on large-scale ASR corpora.
Limitations
- Voice chat still relies on a cascaded system rather than end-to-end.
- Not multilingual; English focus.
- Depends on closed-source models (GPT-4.1, Gemini) for synthetic data generation.
Experimental setup
Datasets: AudioSkills-XL LongAudio-XL AF-Think AF-Chat Speech-in-Sound Caps LibriSpeech GigaSpeech VoxCeleb2 YouTube8M Music4All Million Song Dataset AudioSet
Benchmarks: MMAU MMAR MMSU ClothoAQA AudioCaps Clotho-v2 CompA-R NSynth MusicAVQA MuchoMusic LongAudioBench LibriSpeech
Metrics: accuracy WER CIDEr GPT-4o score human rating (factuality, usefulness, depth) SIM latency
Baselines: Qwen2.5-Omni Qwen2-Audio Gemini 2.5 Pro GPT-4o Audio Audio Flamingo 2 Phi-4-mm R1-AQA Pengi
Builds on: Whisper Audio Flamingo 2 Qwen2.5 LTU GAMA
Closest papers by meaning
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.94)
- Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities (ICML 2024 · similarity 0.91)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.90)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.90)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.89)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.