Research map / Vision-language and multimodal / Cross-modal learning
SonicVisionLM: Playing Sound with Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SonicVisionLM, which uses a VLM (MiniGPT-v2) to describe on-screen sound events from silent video, a timestamp detector, and a time-controllable adapter on Tango to generate time-synchronized sound effects, trained on a new 10k-clip CondPromptBank dataset.
Problem
Generating sound for silent videos is hard because aligning visual and audio representations directly yields incorrect sound meanings, poor timing, and little editability.
Approach
Convert video-to-audio into image-to-text (VLM describes on-screen sounds) plus text-to-audio diffusion, and add a ControlNet-style time-controllable adapter on a frozen text-to-audio latent diffusion model, conditioned on detected sound timestamps, so sound timing is controllable.
Key results
- On Greatest Hits conditional generation, Ours-full raises IoU from 22.4 (CondFoleyGen new) to 39.7 and CLAP-top general to 36.8%.
- On Countix-AV unconditional generation, Ours-small reaches IS 15.1 versus 5.3 for SpecVQGAN, while Ours-full achieves IoU 39.5 on Greatest Hits versus 22.0 for Diff-Foley.
- Subjective study (300 evaluators) gives Ours Time-sync 87 versus 58 for Diff-Foley and 31 for SpecVQGAN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SonicVisionLM-full instead of CondFoleyGen (new) | Greatest Hits conditional generation · Greatest Hits (conditional) | CLAP-top general (%): 16.3 -> 36.8 | Helped |
| Use SonicVisionLM-full instead of CondFoleyGen (new) | Greatest Hits conditional generation · Greatest Hits (conditional) | IoU (%): 22.4 -> 39.7 | Helped |
| Use SonicVisionLM-full instead of CondFoleyGen (new) | Greatest Hits conditional generation · Greatest Hits (conditional) | Onset Acc (%): 19.2 -> 27.6 | Helped |
| Use SonicVisionLM-full instead of CondFoleyGen (new) | Greatest Hits conditional generation · Greatest Hits (conditional) | Onset AP (%): 68.6 -> 78.1 | Helped |
| Use SonicVisionLM-full instead of CondFoleyGen (new) | Greatest Hits conditional generation · Greatest Hits (conditional) | Time Acc (%): 37.4 -> 43.8 | Helped |
| Use SonicVisionLM-full instead of DIFF-FOLEY | Greatest Hits unconditional generation · Greatest Hits (unconditional) | MKL: 5.68 -> 4.67 | Helped |
| Use SonicVisionLM-full instead of DIFF-FOLEY | Greatest Hits unconditional generation · Greatest Hits (unconditional) | FID: 20.0 -> 24.9 | Hurt |
| Use SonicVisionLM-full instead of DIFF-FOLEY | Countix-AV unconditional generation · Countix-AV (unconditional) | MKL: 4.9 -> 9.71 | Hurt |
| Use SonicVisionLM-full instead of DIFF-FOLEY | Countix-AV unconditional generation · Countix-AV (unconditional) | IoU (%): 31.3 -> 42 | Helped |
| Add the time-controllable adapter (ablation) | SonicVisionLM-full · Greatest Hits (time condition ablation) | IoU (%): 39.3 -> 65.7 | Helped |
| Add the time-controllable adapter (ablation) | SonicVisionLM-full · Greatest Hits (time condition ablation) | FD: 44.3 -> 25.4 | Helped |
| Add the time-controllable adapter (ablation) | SonicVisionLM-full · Greatest Hits (time condition ablation) | FAD: 10.87 -> 3.19 | Helped |
Surprising findings
- Ours-full has a lower IS (3.26) than Diff-Foley (3.84) and SpecVQGAN on Greatest Hits's unconditional setting, and worse MKL/FID on Countix-AV than baselines, which the authors attribute to deliberately diverging from ground truth.
Limitations
- Visual understanding and timestamp detection need refinement, per the authors.
- Audio diversity and range of generation control remain limited.
- Unconditional MKL and FID are worse than baselines on Countix-AV.
Experimental setup
Datasets: CondPromptBank Greatest Hits Countix-AV
Benchmarks: Greatest Hits Countix-AV
Metrics: CLAP-top Onset Acc Onset AP Time Acc IoU MKL FID IS FD FAD
Baselines: CondFoleyGen SpecVQGAN Diff-Foley Tango
Builds on: Tango ControlNet CondFoleyGen MiniGPT-v2
Closest papers by meaning
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.88)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.88)
- Sonic: Shifting Focus to Global Audio Perception in Portrait Animation (CVPR 2025 · similarity 0.87)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.87)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.