Research map / Vision-language and multimodal / Multimodal LLMs
ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
ThinkSound uses a fine-tuned VideoLLaMA2 to produce chain-of-thought reasoning that guides a flow-matching MM-DiT audio foundation model through three stages (foley generation, click-based object refinement, instruction-based editing), and releases the AudioCoT dataset.
Problem
End-to-end video-to-audio generation struggles to capture fine visual dynamics, temporal relations, and user control, and prior MLLM-based V2A pipelines either lose visual detail via captions or fragment the process into separate models.
Approach
Fine-tune an MLLM on a CoT-annotated audio dataset to produce stepwise reasoning that conditions a single unified flow-matching audio model across generation, object-centric refinement, and text-guided editing, with classifier-free guidance dropout over modalities and gated video fusion in an MM-DiT.
Key results
- On VGGSound, ThinkSound reports FD 34.56 and CLAP_CoT 0.46, versus 43.26 FD and 0.40 CLAP_CoT for MMAudio.
- Subjective MOS-Q 4.02 and MOS-A 4.18 on VGGSound, above MMAudio's 3.84 and 3.97.
- On the out-of-distribution MovieGen Audio Bench, CLAP_CoT reaches 0.51 versus 0.47 for MovieGen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ThinkSound instead of MMAudio | VGGSound video-to-audio generation · VGGSound test set | KL PaSST: 1.65 -> 1.52 | Helped |
| use ThinkSound instead of MovieGen | Out-of-distribution video-to-audio generation · MovieGen Audio Bench | CLAP CoT: 0.47 -> 0.51 | Helped |
| add CoT reasoning (ablation) | ThinkSound · VGGSound test set | CLAP CoT: 0.41 -> 0.46 | Helped |
| add CoT reasoning (ablation) | ThinkSound object-focused generation · Object-focused generation | CLAP: 0.43 -> 0.48 | Helped |
| add CoT reasoning (ablation) | ThinkSound audio editing · AudioCoT test set | FD: 45.78 -> 34.78 | Helped |
| combine CLIP caption features with T5 CoT features (ablation) | ThinkSound text encoding · VGGSound test set | KL PaSST: 1.54 to 1.52 | Helped |
| add video features using linear integration (ablation) | ThinkSound video-audio integration · VGGSound test set | DeSync: 0.50 to 0.46 | Helped |
| replace linear video integration with gated video fusion (ablation) | ThinkSound video-audio integration · VGGSound test set | FD: 38.96 -> 34.56 | Helped |
| increase model size from Small to Medium to Large (ablation) | ThinkSound · VGGSound test set | FD: 43.26 -> 37.62 -> 34.56 | Helped |
| use fine-grained rather than coarse-grained CoT (ablation) | ThinkSound · VGGSound test set | CLAP CoT: 0.34 -> 0.46 | Helped |
| preserve the ordered logical structure of CoT rather than shuffling its sentences (ablation) | ThinkSound · VGGSound test set | FD: 40.52 -> 34.56 | Helped |
| include reasoning structure rather than providing keywords only (ablation) | ThinkSound · VGGSound test set | FD: 39.35 -> 34.56 | Helped |
Surprising findings
- Ground-truth VGGSound captions get a low CLAP_cap score (0.28), so the authors switch to CoT-based CLAP as the primary alignment metric.
Limitations
- MLLMs struggle with precise temporal and spatial localization of sound events in video.
- Open video-audio datasets have limited diversity and may miss rare or culturally specific sounds.
- Subjective evaluation relies on a small number of raters and sample subsets.
Experimental setup
Datasets: VGGSound AudioSet AudioSet-SL Freesound AudioCaps BBC Sound Effects MovieGen Audio Bench
Benchmarks: VGGSound test set MovieGen Audio Bench AudioCoT test set
Metrics: FD KL_PaSST KL_PaNNs DeSync CLAP_cap CLAP_CoT MOS-Q MOS-A
Baselines: See&Hear V-AURA FoleyCrafter Frieren V2A-Mapper MMAudio MovieGen AudioLDM-2 Edit Friendly DDPM
Builds on: VideoLLaMA2 MMAudio flow matching MM-DiT Grounded SAM2
Closest papers by meaning
- PrismAudio: Decomposed Chain-of-Thought and Multi-dimensional Rewards for Video-to-Audio Generation (ICLR 2026 · similarity 0.90)
- Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts (NeurIPS 2026 · similarity 0.88)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.88)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.88)
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.88)
- SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.