Research map / Large language models / AI and society
Listen, Think, and Understand
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LTU, an audio-LLM that couples an AST audio encoder with LLaMA-7B via LoRA, trained with a perception-to-understanding curriculum on the 5.6M-pair OpenAQA dataset, for both closed-ended audio perception and open-ended audio reasoning.
Problem
Existing audio models map sound to fixed label sets and lack reasoning or understanding beyond perception, while LLMs lack audio perception; the paper asks whether one model can have both.
Approach
Bridge a pretrained audio encoder (AST) to a frozen LLaMA-7B with a projection layer and LoRA adapters, trained on GPT-3.5-generated open-ended audio QA from metadata (Audio Instruction Generation) plus rule-based closed-ended QA, using a curriculum that moves from closed-ended perception tasks to open-ended understanding.
Key results
- LTU outperforms CLAP on all eight classification benchmarks with an average relative improvement of 23.6%, and reaches 17.0 and 11.9 SPICE on AudioCaps and Clotho.
- Human evaluation on AMT: instruction following and factual correctness rate of 82.9% for human-written questions and 75.9% for GPT-4-generated questions.
- Removing the perception-to-understanding curriculum drops average classification performance from 50.3 to 23.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove LoRA adapters (ablation) | LTU · Audio classification average | average performance: 50.3 -> 38.3 | Hurt |
| change LoRA rank from 8 to 4 (ablation) | LTU · Audio classification average | average performance: 50.3 -> 50.2 | No clear effect |
| change LoRA rank from 8 to 16 (ablation) | LTU · Audio classification average | average performance: 50.3 -> 50.5 | No clear effect |
| fully fine-tune LLaMA instead of using LoRA (ablation) | LTU · Audio classification average | average performance: 50.3 -> 51.3 | Helped |
| fully fine-tune LLaMA instead of using LoRA (ablation) | LTU · Pure language tasks | instruction following rate: 87.6 -> 53.5 | Hurt |
| freeze the audio branch (ablation) | LTU · Audio classification average | average performance: 50.3 -> 33.7 | Hurt |
| remove the perception-to-understanding curriculum (ablation) | LTU · Audio classification average | average performance: 50.3 -> 23.0 | Hurt |
| remove open-ended training (ablation) | LTU · Audio classification average | average performance: 50.3 -> 37.1 | Hurt |
| remove open-ended training while keeping total iterations the same (ablation) | LTU · Audio classification average | average performance: 50.3 -> 47.3 | Hurt |
| include unanswerable questions in training (ablation) | LTU · Test unanswerable questions | refusal rate: 51.9% -> 69.3% | Helped |
| include unanswerable questions in training (ablation) | LTU · Audio classification average | average performance: 50.0 -> 50.3 | Helped |
| increase LLaMA size from 7B to 13B (ablation) | LTU · Audio classification average | average performance: 50.3 -> 48.9 | Hurt |
Surprising findings
- Training with unanswerable questions raises refusal rate to unanswerable questions from 51.9% to 69.3%, even though such questions are only about 6.5% of data.
- Full fine-tuning of LLaMA at the learning rate of 2e-5 slightly improves audio tasks but causes severe loss of language instruction following (53.5% vs 87.6%).
Limitations
- Uses only the 7B LLaMA model because of compute limits; the 13B variant did not clearly improve audio tasks.
- Audio is temporally downsampled to 3.2 Hz, which may limit fine-grained temporal reasoning.
- Limited speech understanding; not an ASR model.
Experimental setup
Datasets: AudioSet AudioSet-Strong VGGSound FSD50K AudioCaps FreeSound Clotho v2 Sound Bible
Benchmarks: ESC50 DCASE VGGSound TUT Beijing Opera VocalSound FSD50K AudioSet AudioCaps Clotho
Metrics: accuracy micro F1 mAP SPICE instruction following rate factual correctness rate
Baselines: CLAP AudioCLIP Pengi LLaVA AST
Builds on: LLaMA Vicuna AST CAV-MAE LoRA
Closest papers by meaning
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.93)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.92)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.91)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.91)
- AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound (NeurIPS 2025 · similarity 0.89)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.