Research map / Large language models / Language, safety and interpretability
ADIFF: Explaining audio difference using natural language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the audio difference explanation task with two LLM-generated, three-tier datasets (ACD, CLD) and proposes ADIFF, a prefix-tuned GPT2 model with cross-projection, position captioning, and three-stage training that outperforms a naive baseline and Qwen-Audio.
Problem
Explaining in natural language how two audio recordings differ (events, scenes, signal characteristics, listener emotion) has not been studied, and existing audio-language models lack comparative reasoning over two audio inputs.
Approach
A prefix-tuning audio-language model that projects two audio embeddings separated by a separator token through a cross-projection transformer, trained with position captioning and a three-stage (unimodal pretraining, frozen-LM grounding, LM fine-tuning) schedule to produce tiered difference explanations.
Key results
- On CLD Tier 1, ADIFF reaches SPIDEr 0.652 versus 0.212 for the naive baseline and 0.230 for QwenAC (LoRA).
- In human evaluation averaged across Studio, FSD50K and GTZAN, ADIFF scores 3.47 correctness, 3.53 granularity and 3.57 readability, above Qwen-AC full fine-tuning (3.28/3.45/3.49).
- ADIFF uses a 128M-parameter GPT-2 decoder yet beats the 7B Qwen-Audio on most objective metrics, though Qwen-AC full fine-tuning wins on CLD Tier 2 and some human-rated dimensions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add separator token and cross-projection to the baseline (ablation) | ADIFF · ACD, all tiers | SPIDEr: 0.220 -> 0.287 (Tier 1); 0.225 -> 0.243 (Tier 2); 0.123 -> 0.154 (Tier 3) | Helped |
| add separator token and cross-projection to the baseline (ablation) | ADIFF · CLD, all tiers | SPIDEr: 0.212 -> 0.195 (Tier 1); 0.641 -> 0.904 (Tier 2); 0.166 -> 0.196 (Tier 3) | Mixed |
| use pretrained HTSAT weights instead of random frozen audio-encoder weights (ablation) | Baseline architecture · ACD, all tiers | SPIDEr: 0.128 -> 0.220 (Tier 1); 0.154 -> 0.225 (Tier 2); 0.107 -> 0.123 (Tier 3) | Helped |
| use pretrained HTSAT weights instead of random frozen audio-encoder weights (ablation) | Baseline architecture · CLD, all tiers | SPIDEr: 0.127 -> 0.212 (Tier 1); 0.589 -> 0.641 (Tier 2); 0.127 -> 0.166 (Tier 3) | Helped |
| scale the decoder from 128M to larger language models under the same compute budget (ablation) | ADIFF · ACD and CLD | average score across tiers | Hurt |
| add position-based audio captioning data (ablation) | ADIFF with cross-projection · ACD, all tiers | average score across tiers | Helped |
| add position-based audio captioning data (ablation) | ADIFF with cross-projection · CLD, all tiers | average score across tiers | Mixed |
| fine-tune the language model in stage 3 (ablation) | ADIFF with position captioning · ACD and CLD, all tiers | average score across tiers | Helped |
| use ADIFF instead of the strongest listed QwenAC baseline | Audio difference explanation · ACD Tier 1 | SPIDEr: 0.258 -> 0.303 | Helped |
| use ADIFF instead of the strongest listed QwenAC baseline | Audio difference explanation · CLD Tier 2 | SPIDEr: 0.958 -> 0.692 | Hurt |
| use ADIFF instead of the naive baseline | Audio difference explanation · ACD and CLD, all tiers | human evaluation granularity: 3.35 -> 3.53 (average) | Helped |
| use ADIFF instead of QwenAC (F) | Audio difference explanation · Studio readability | Readability: 3.37 -> 3.34 | Hurt |
Surprising findings
- A language-only model with a randomly initialized, frozen audio encoder achieves competitive scores, showing that metrics can be inflated by linguistic patterns alone.
- Scaling GPT-2 from 128M to 1.5B under equal compute budget did not improve performance; larger models needed more epochs to benefit.
Limitations
- Explanations generated by an LLM and only test-set verified by humans, so training labels may contain hallucinations.
- Small 128M language model limits fluency and readability relative to 7B ALMs.
- Objective metrics rely on linguistic biases and may penalize diverse outputs.
Experimental setup
Datasets: AudioCaps Clotho v2.1 AudioSet FSD50K GTZAN
Benchmarks: ACD CLD
Metrics: BLEU-4 METEOR SPIDEr SPICE CIDEr correctness granularity readability
Baselines: Baseline (naive prefix-tuning) Qwen-Audio (QwenAC) zero-shot Qwen-Audio LoRA Qwen-Audio full fine-tuning
Builds on: prefix tuning HTSAT GPT-2 Pengi
Closest papers by meaning
- AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound (NeurIPS 2025 · similarity 0.87)
- CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models (ICLR 2024 · similarity 0.87)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.87)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.86)
- Audio Entailment: Assessing Deductive Reasoning for Audio Understanding (AAAI 2025 · similarity 0.86)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.