Research map / Large language models / Language, safety and interpretability
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds SALMONN, a speech-audio-music LLM that couples Whisper and BEATs encoders via a window-level Q-Former and LoRA on Vicuna-13B, and adds a few-shot activation tuning stage to recover emergent cross-modal abilities lost to task over-fitting.
Problem
Instruction-tuned audio-language LLMs overfit to trained tasks (e.g., ASR and audio captioning) and lose the emergent zero-shot cross-modal abilities of the underlying text LLM, failing to follow instructions for untrained tasks.
Approach
Dual auditory encoders (Whisper + BEATs) feed a window-level Q-Former into a LoRA-adapted Vicuna; reducing the LoRA scaling factor reveals an intrinsic conditional LM that can be reactivated by a short few-shot activation tuning stage on self-generated story responses.
Key results
- With activation tuning, SALMONN keeps ASR near reference (2.1/4.9/10.0 WER) and reaches 82.57 story diversity with 1.00 following rate versus 0.00 without activation tuning.
- Activation tuning raises SQQA following-rate-adjusted accuracy from 0.19 (0.29) to 0.41 (0.98) and SAC from 0.02 to 0.50.
- Reducing the LoRA scaling factor to about 2.0 at test time makes the model suddenly show cross-modal reasoning abilities without any extra training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add activation tuning using stories (ablation) | SALMONN · WikiQA | Accuracy (FR): 0.19 (0.29) -> 0.41 (0.98) | Helped |
| Add activation tuning using stories (ablation) | SALMONN · AudioCaps | Diversity (FR): 7.77 (0.00) -> 82.57 (1.00) | Helped |
| Add activation tuning using stories (ablation) | SALMONN · In-house Data | Accuracy (FR): 0.02 (0.04) -> 0.50 (0.73) | Helped |
| Add activation tuning using stories (ablation) | SALMONN · LibriMix | WER: 20.7 -> 23.0 | Hurt |
| Add activation tuning using stories (ablation) | SALMONN · AudioCaps | METEOR: 25.6 -> 24.0 | Hurt |
| Use long-answer QA instead of stories for activation tuning (ablation) | SALMONN · AudioCaps | Diversity (FR): 82.57 -> 59.82 | Hurt |
| Use long-answer QA instead of stories for activation tuning (ablation) | SALMONN · In-house Data | Accuracy (FR): 0.50 (0.73) -> 0.34 (0.71) | Hurt |
| Train on ASR with long labels instead of stories for activation tuning (ablation) | SALMONN · AudioCaps | Diversity (FR): 82.57 -> 7.87 | Hurt |
| Train on text-only stories, without audio input (ablation) | SALMONN · AudioCaps | Diversity (FR): 82.57 -> 8.45 | Hurt |
| Fine-tune LoRA only, without tuning Q-Former, for story activation (ablation) | SALMONN · In-house Data | Accuracy (FR): 0.50 (0.73) -> 0.34 (0.65) | Hurt |
| Reduce LoRA scaling factor at test time instead of activation tuning (ablation) | SALMONN · speech/audio cross-modal reasoning | SQQA, Story, and SAC performance | Mixed |
Surprising findings
- Training on ASR with long transcriptions or text-only story fine-tuning fails to activate emergent abilities, while story or long-QA responses from the audio succeed.
- Reducing the LoRA scaling factor at test time reactivates cross-modal reasoning without additional training.
Limitations
- Phoneme recognition and overlapped speech recognition lag specialised systems; OSR WER 23.0 vs reference 7.6.
- SQQA performance drops, suggesting LoRA adaptation causes forgetting of text-only commonsense knowledge.
- Level 2 and 3 reference values are cascaded Whisper+Vicuna outputs or none, and Story/SAC metrics are heuristic or GPT-3.5-judged.
Experimental setup
Datasets: LibriSpeech GigaSpeech WavCaps AudioCaps Clotho CoVoST2-En2Zh IEMOCAP MusicCaps LibriMix VoxCeleb1 MillionSong MusicNet
Benchmarks: LibriSpeech GigaSpeech CoVoST2 AudioCaps LibriSpeech test-clean (PR) IEMOCAP MusicCaps LibriMix VoxCeleb1 Inspec WikiQA SLURP
Metrics: WER BLEU4 METEOR SPIDEr PER accuracy RougeL following rate diversity
Baselines: Whisper + Vicuna cascade AudioGPT SALMONN w/o activation tuning
Builds on: Vicuna Whisper BEATs Q-Former LoRA
Closest papers by meaning
- Listen, Think, and Understand (ICLR 2024 · similarity 0.89)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.87)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.87)
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models (NeurIPS 2025 · similarity 0.86)
- AND: Audio Network Dissection for Interpreting Deep Acoustic Models (ICML 2024 · similarity 0.86)
- SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.