Research map / Video / Video generation
Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Foley-Flow, a video-to-audio generator that aligns unimodal AV encoders via masked audio reconstruction guided by video segments, then generates audio with a dynamic conditional velocity flow conditioned on time-varying video features.
Problem
Video-to-audio generation needs both semantic and rhythmic (temporal) audio-visual synchronization, but contrastive AV alignment and global video conditioning capture only overall semantics and miss segment-level rhythm.
Approach
Replace contrastive AV alignment with masked audio segment reconstruction conditioned on temporally aligned video features (VAMA), and use temporally varying video features as dynamic per-segment conditions in a velocity/variance-preserving flow (GVAF) for audio generation.
Key results
- Foley-Flow reaches KLD 0.97, FAD 0.52, Align Acc 98.97 on VGGSound, versus the best prior VATT at KLD 2.25 and FoleyGen at FAD 2.59.
- Ablation: removing VAMA raises KLD to 1.92 and drops Align Acc to 93.86; removing GVAF raises FAD to 1.57.
- Using EVA-CLIP video and AudioMAE audio encoders gives the best results among tested encoder combinations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Foley-Flow instead of VATT | Video-to-audio generation · VGGSound test | KLD: 2.25 -> 0.97 | Helped |
| Use Foley-Flow instead of V2A-Mapper | Video-to-audio generation · VGGSound test | FAD: 0.99 -> 0.52 | Helped |
| Use Foley-Flow instead of VATT | Video-to-audio generation · VGGSound test | Align Acc: 82.81 -> 98.97 | Helped |
| Add VAMA (ablation) | Foley-Flow without VAMA (GVAF only) · VGGSound test | KLD: 1.92 -> 0.97 | Helped |
| Add VAMA (ablation) | Foley-Flow without VAMA (GVAF only) · VGGSound test | Align Acc: 93.86 -> 98.97 | Helped |
| Add GVAF (ablation) | Foley-Flow without GVAF (VAMA only) · VGGSound test | FAD: 1.57 -> 0.52 | Helped |
| Add GVAF (ablation) | Foley-Flow without GVAF (VAMA only) · VGGSound test | Align Acc: 96.29 -> 98.97 | Helped |
| Use GDCF instead of DCF (ablation) | Foley-Flow · VGGSound test | KLD: 1.28 -> 0.97 | Helped |
| Use GDCF instead of DCF (ablation) | Foley-Flow · VGGSound test | FAD: 0.85 -> 0.52 | Helped |
| Use GDCF instead of DCF (ablation) | Foley-Flow · VGGSound test | Align Acc: 97.73% -> 98.97% | Helped |
| Set masking ratio to 0.8 instead of 0.3, 0.5, 0.7, or 0.9 (ablation) | Foley-Flow VAMA · VGGSound test | KLD: 1.25, 1.16, 1.09, 1.02 -> 0.97 | Helped |
| Set masking ratio to 0.8 instead of 0.3, 0.5, 0.7, or 0.9 (ablation) | Foley-Flow VAMA · VGGSound test | FAD: 1.06, 0.87, 0.65, 0.56 -> 0.52 | Helped |
Surprising findings
- Reported gains over prior work are very large (e.g., FAD 0.52 vs 0.99 for V2A-Mapper, and Align Acc 98.97 vs 82.81 for VATT), while the paper gives few implementation details for the flow model and training.
Limitations
- Flow formulation and training details (architecture, objective specifics) are vague in the text.
- Evaluation limited to VGGSound test set in the main table; AudioSet is mentioned but not reported.
- Generation speed metric is defined but no numbers are reported.
Experimental setup
Datasets: VGGSound AudioSet
Benchmarks: VGGSound
Metrics: KLD FAD Align Acc generation speed
Baselines: SpecVQGAN Im2Wav Diff-Foley FoleyGen V2A-Mapper Seeing & Hearing MaskVAT VAB VATT
Builds on: masked autoencoders velocity flow Diff-Foley CAV-MAE
Closest papers by meaning
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.91)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.90)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.89)
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.88)
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.88)
- Audio-Sync Video Generation with Multi-Stream Temporal Control (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.