Research map / Video / Video generation
AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer
Pengjun Fang, Yingqing He, Yazhou Xing, Qifeng Chen, Ser-Nam Lim, Harry Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| switch from overlapping (stage I) to non-overlapping conditioning in stage II (ablation) | AC-Foley · VGGSound test | FD_PaSST: 80.07 -> 56.00 | Improved |
| add stage II non-overlap training on top of non-overlap-only baseline (two-stage vs non-overlap only) (ablation) | AC-Foley · VGGSound test | FD_PaSST: 60.82 -> 56.00 | Improved |
| add finetuning on high audio-visual correspondence VGGSound subset (ablation) | AC-Foley · VGGSound test | Onset Acc.: 0.2599 -> 0.2832 | Improved |
| add subset finetuning (ablation) | AC-Foley · VGGSound test | MCD: 11.37 -> 11.37 | No clear effect |
| use average pooling instead of attention-based pooling for conditional audio (ablation) | AC-Foley · VGGSound test | Onset Acc.: 0.2598 -> 0.2832 | No clear effect |
| add sync features (ablation) | AC-Foley · VGGSound test | DeSync: 1.240 -> 0.465 | Improved |
| add reference audio conditioning (ablation) | AC-Foley · VGGSound test | FD_PaSST: 64.90 -> 56.00 | Improved |
| add reference audio conditioning (ablation) | AC-Foley · VGGSound test | DeSync: 0.410 -> 0.465 | Hurt |
| add text conditioning (ablation) | AC-Foley · VGGSound test | FD_PaSST: 55.63 -> 56.00 | No clear effect |
| add reference audio condition vs null embedding (ablation) | AC-Foley · VGGSound test, 5 reference clips | MCD: 20.95 -> 18.24; 16.12 -> 11.96; 15.56 -> 14.43; 22.74 -> 12.20; 15.83 -> 10.85 | Improved |
| use VAE-encoded reference audio instead of CLAP-encoded (MMAudio + Clap) | video-to-audio with audio conditioning · VGGSound test | MCD: 14.63 -> 11.37 | Improved |
Closest papers by meaning
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.88)
- Video-Guided Foley Sound Generation with Multimodal Controls (CVPR 2025 · similarity 0.88)
- V2Meow: Meowing to the Visual Beat via Video-to-Music Generation (AAAI 2024 · similarity 0.88)
- Tri-Ergon: Fine-Grained Video-to-Audio Generation with Multi-Modal Conditions and LUFS Control (AAAI 2025 · similarity 0.88)
- Creative Text-to-Audio Generation via Synthesizer Programming (ICML 2024 · similarity 0.87)
- SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
