Research map / Video / Video generation
Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
Simian Luo, Chuanhao Yan, Chenxu Hu, Hang Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CAVP visual features instead of ResNet50 (ablation) | Diff-Foley · VGGSound | Align Acc (%): 54.88 -> 92.35 | Improved |
| use CAVP visual features instead of CLIP (ablation) | Diff-Foley · VGGSound | Align Acc (%): 79.23 -> 92.35 | Improved |
| use CAVP features instead of CLIP (ablation) | Diff-Foley · VGGSound | Inception Score: 57.43 -> 52.07 | Hurt |
| add Temporal Split & Merge Augmentation (ablation) | Diff-Foley · VGGSound | Align Acc (%): 79.79 -> 92.35 | Improved |
| increase model size from 335M to 859M (no pretrain) (ablation) | Diff-Foley · VGGSound | Inception Score: 16.96 -> 24.91 | Improved |
| initialize with pretrained SD-V1.4 weights (ablation) | Diff-Foley-L · VGGSound | Inception Score: 24.91 -> 52.07 | Improved |
| add classifier-free guidance (ablation) | Diff-Foley · VGGSound | Align Acc (%): 69.41 -> 92.35 | Improved |
| add classifier guidance alone (no CFG) (ablation) | Diff-Foley · VGGSound | Align Acc (%): 69.41 -> 66.05 | Hurt |
| add classifier guidance on top of CFG (double guidance) (ablation) | Diff-Foley · VGGSound | Align Acc (%): 92.35 -> 94.78 | Improved |
| pretrain CAVP on VGGSound plus AudioSet-V2A instead of VGGSound only (ablation) | Diff-Foley · VGGSound | Align Acc (%): 85.88 -> 92.35 | Improved |
| use Diff-Foley instead of Im2Wav | video-to-audio · VGGSound | Align Acc (%): 67.40 -> 94.05 | Improved |
| use Diff-Foley instead of SpecVQGAN/Im2Wav on distribution metrics | video-to-audio · VGGSound | FID/KL: FID 8.93 (SpecVQGAN) vs 9.87; KL 5.20 (Im2Wav) vs 6.43 | Mixed |
Later papers that compared with this method
11 later accepted papers report Diff-Foley as a baseline in their results tables; 11 of them report a better number on most of the results they share with it.
- FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds (IJCV 2026 · better on 8 of 8 shared results)
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · better on 1 of 1 shared result)
- MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation (ICLR 2025 · better on 4 of 4 shared results)
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · better on 3 of 3 shared results)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · better on 2 of 2 shared results)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · better on 1 of 1 shared result)
- VAFlow: Video-to-Audio Generation with Cross-Modality Flow Matching (ICCV 2025 · better on 1 of 1 shared result)
- TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesis (ICCV 2025 · better on 1 of 1 shared result)
3 more not listed.
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.90)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.90)
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.89)
- Tri-Ergon: Fine-Grained Video-to-Audio Generation with Multi-Modal Conditions and LUFS Control (AAAI 2025 · similarity 0.89)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.88)
- Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
