Research map / Large language models / Language, safety and interpretability
Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Audio Flamingo, a Flamingo-style audio language model with sliding-window CLAP-based audio features, gated cross-attention, retrieval-based in-context learning, and multi-turn dialogue fine-tuning.
Problem
Existing LLM-based audio understanding models lack in-context few-shot adaptation to unseen tasks, strong multi-turn dialogue ability, and efficient handling of variable-length audio.
Approach
Combines a sliding-window ClapCap audio encoder with Flamingo-style gated cross-attention (linear in audio tokens), trains on interleaved ICL samples retrieved by CLAP kNN with a block upper-triangular cross-attention mask, and fine-tunes on GPT-4-generated multi-turn audio dialogue datasets.
Key results
- Achieves CIDEr 0.465 on Clotho-v2 (vs previous SOTA 0.441) and 0.502 zero-shot on AudioCaps (vs 0.281).
- 8-shot retrieval-augmented audio captioning on AudioCaps reaches CIDEr 0.538 (16-shot 0.546) vs RECAP 0.359 at 4 shots.
- On multi-turn dialogue (AF-Dialogue-AudioSetSL), fine-tuned model reaches CIDEr 1.622 vs LTU fine-tuned 0.823.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ICL-based RAG with 8 examples | Audio Flamingo · CREMA-D, Ravdess, US8K, GTZAN, Medley-solos-DB | ACC: +5.3%; +14.3%; +19.4%; +11.6%; +3.0% | Helped |
| Add ICL-based RAG with 16 examples | Audio Flamingo · CREMA-D, Ravdess, US8K, GTZAN, Medley-solos-DB | ACC: +8.9%; +19.3%; +16.9%; +8.4%; +3.3% | Helped |
| Increase ICL examples from 8 to 16 (ablation) | Audio Flamingo · CREMA-D, Ravdess, US8K, GTZAN, Medley-solos-DB | ACC | Mixed |
| Use ICL-based RAG for unseen-label classification | Audio Flamingo · BG-Gun-Sound | accuracy: 1.6% -> 53.5% | Helped |
| Use ICL-based RAG for unseen-label classification | Audio Flamingo · Birds-Dataset-Till-2022 | accuracy: 0.0% -> 42.2% | Helped |
| Use ICL-based retrieval for audio captioning | Audio Flamingo · AudioCaps | CIDEr: 0.359 -> 0.518 | Helped |
| Fine-tune on generated dialogue datasets | Audio Flamingo chat · AF-Dialogue-AudioSetSL test | CIDEr: 0.823 -> 1.622 | Helped |
| Fine-tune on generated dialogue datasets | Audio Flamingo chat · AF-Dialogue-MusicCaps test | CIDEr: 0.419 -> 1.143 | Helped |
| Use instruction-tuned OPT-IML-MAX-1.3B instead of opt-1.3b (ablation) | Audio Flamingo · Audio understanding tasks | relative performance | Helped |
| Replace ClapCap with LAION-CLAP audio encoder (ablation) | Audio Flamingo · Audio understanding tasks | relative performance | Hurt |
| Replace ClapCap with Clap2023 audio encoder (ablation) | Audio Flamingo · Close-ended tasks and open-ended tasks | relative performance | Mixed |
| Increase pretraining and SFT data amount (ablation) | Audio Flamingo · Audio understanding benchmarks | evaluation performance | Helped |
Surprising findings
- Audio Flamingo trails prior SOTA on several tasks, e.g. CochlScene (83.0% vs 91.6%) and MusicAVQA audio-only (71.6% vs 72.1%).
- The model ignores noisy retrieved samples and predicts correctly (e.g., flute classification with three clarinet retrievals).
Limitations
- Uses a small 1.3B OPT-IML language model; authors suggest larger LMs could help.
- Focuses on non-speech sound, music, and non-verbal speech; complex speech tasks beyond transcription are not covered.
- Some baselines outperform the model on certain in-distribution tasks.
Experimental setup
Datasets: WavCaps Macs SoundDescs Clotho-v2 WavText5K LAION-630k Clotho-AQA Open-AQA AudioSet FSD50k CochlScene NonSpeech7K
Benchmarks: Clotho-v2 ClothoAQA MusicAVQA CochlScene NonSpeech7k FSD50k NSynth AudioCaps CREMA-D Ravdess US8K GTZAN
Metrics: CIDEr accuracy F1 Bleu4 Rouge-L
Baselines: Qwen-Audio LTU MU-LLaMA Pengi SALMONN RECAP
Builds on: Flamingo ClapCap OPT-IML LAION-CLAP RECAP
Closest papers by meaning
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.92)
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models (NeurIPS 2025 · similarity 0.91)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.88)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.87)
- UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner (NeurIPS 2024 · similarity 0.86)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.