Research map / Video / Video generation
Video-Guided Foley Sound Generation with Multimodal Controls
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MultiFoley, a diffusion-transformer model for video-guided Foley generation with text, audio, and video conditioning, trained jointly on VGGSound and a proprietary high-quality SFX dataset to produce 48kHz audio.
Problem
Video-to-audio generation for Foley lacks user control and produces low-bandwidth, poorly synchronized audio because internet video soundtracks are noisy and misaligned with visuals.
Approach
Jointly train a diffusion transformer on VGGSound (audio-visual-text) and a licensed high-quality SFX text-audio library, using quality tags as a bridge, and condition on text (with negative prompting), reference audio-visual segments (audio extension), and silent video to enable flexible multimodal Foley control at 48kHz.
Key results
- On VGGSound video-to-audio, MultiFoley achieves CLAP 34.4% and AV-Sync 0.80s, best among compared methods, and FAD@AUD 4.62 with KLD 1.43.
- In a 2AFC human study against FoleyCrafter, MultiFoley wins 90.2% overall and 94.5% on synchronization.
- Using a 'high quality' tag with 48kHz generation improves FAD@AUD from 4.62 to 4.09 while FAD@VGG worsens from 2.92 to 4.37.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MultiFoley for video-to-audio generation instead of the strongest baseline on CLAP, FoleyCrafter | Video-to-audio generation · VGGSound test | CLAP: 25.3 -> 34.4 | Helped |
| Use MultiFoley instead of Frieren | Video-to-audio generation · VGGSound test | AV-Sync: 0.87 -> 0.80 | Helped |
| Use MultiFoley instead of the best baseline on FAD@AUD, FoleyGen | Video-to-audio generation · VGGSound test | FAD@AUD: 4.62 vs 4.62 | No clear effect |
| Add negative prompting (ablation) | MultiFoley text-controlled Foley generation · VGGSound-sync text-control evaluation | CLAP accuracy: 85.5 -> 93.2 | Helped |
| Add negative prompting (ablation) | MultiFoley text-controlled Foley generation · VGGSound-sync text-control evaluation | AV-Sync: 0.81 -> 0.93 | Hurt |
| Add subset fine-tuning on better-aligned VGGSound examples (ablation) | MultiFoley · VGGSound test | ImageBind: 27.3 -> 28.0 | Helped |
| Add subset fine-tuning on better-aligned VGGSound examples (ablation) | MultiFoley · VGGSound test | AV-Sync: 0.81 -> 0.80 | Helped |
| Use the high quality tag instead of the low quality tag (ablation) | MultiFoley · VGGSound test | FAD@AUD: 4.62 -> 4.09 | Helped |
| Use the high quality tag instead of the low quality tag (ablation) | MultiFoley · VGGSound test | FAD@VGG: 2.92 -> 4.37 | Hurt |
| Remove text conditioning (ablation) | MultiFoley · VGGSound test | ImageBind: 28.0 -> 22.4 | Hurt |
| Remove text conditioning (ablation) | MultiFoley · VGGSound test | AV-Sync: 0.80 -> 0.77 | Helped |
| Add audio conditioning to video and text conditioning | MultiFoley Foley extension · VGGSound-Sync | CLAP: 55.4 -> 59.8 | Helped |
Surprising findings
- Without any text input, synchronization metrics remain high, suggesting temporal alignment is largely driven by video while semantics rely on text.
- Standard VGGSound audio downloaded from YouTube is often band-limited to 32kHz or lower, so models trained only on it generate low-bandwidth audio.
Limitations
- Trained on a small-scale in-the-wild dataset (VGGSound), which constrains capability.
- Struggles with multiple sound events alongside text, causing confusion about event timing.
- The HQ-SFX training dataset is proprietary, limiting reproducibility.
Experimental setup
Datasets: VGGSound HQ-SFX VGGSound-Sync Greatest-Hits Adobe SFX Audition
Benchmarks: VGGSound test set VGGSound-Sync Greatest-Hits
Metrics: FAD@VGG FAD@AUD KLD ImageBind score CLAP score AV-Sync human win rate
Baselines: Im2Wav V2A-mapper Diff-Foley FoleyGen VAB Frieren FoleyCrafter
Builds on: latent diffusion models Diffusion Transformer DAC-VAE CAVP ImageBind
Closest papers by meaning
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.88)
- Tri-Ergon: Fine-Grained Video-to-Audio Generation with Multi-Modal Conditions and LUFS Control (AAAI 2025 · similarity 0.85)
- SonicVisionLM: Playing Sound with Vision Language Models (CVPR 2024 · similarity 0.85)
- Animate and Sound an Image (CVPR 2025 · similarity 0.84)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.84)
- Audio-Sync Video Generation with Multi-Stream Temporal Control (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.