Research map / Video / Video generation
Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frieren is a non-autoregressive video-to-audio model that uses rectified flow matching with channel-level audio-visual fusion, reflow and guided one-step distillation to generate synchronized audio in one to 25 steps.
Problem
Video-to-audio generation needs high audio quality, strong audio-visual temporal synchrony, and fast sampling, which prior autoregressive and diffusion-based V2A models balance poorly.
Approach
Apply rectified flow matching to spectrogram-latent V2A generation with a feed-forward transformer estimator that fuses visual and audio features channel-wise without temporal downsampling, then use reflow with CFG-guided vector fields and one-step self-distillation to get few-step sampling.
Key results
- Frieren reaches 97.22% alignment accuracy and IS 12.42 at 25 Euler steps on VGGSound, versus 95.03% and 11.11 for Diff-Foley (CG), a 6.2% IS gain over the strong diffusion baseline.
- With reflow and one-step distillation, Frieren reaches 97.85% accuracy in one step, 9.3x faster than 25-step sampling (0.031 s vs 0.288 s per sample).
- Frieren's inference is 7.3x faster than Diff-Foley at 25 steps (0.288 s vs 2.104 s on RTX-4090).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use rectified flow matching instead of the diffusion prediction objective | Frieren vs LDM · VGGSound test | IS: 10.09 -> 12.42 | Helped |
| use reflow for 1-step sampling (ablation) | Frieren · VGGSound test, 1 step | Acc (%): 43.18 -> 94.96 | Helped |
| add one-step distillation after reflow (ablation) | Frieren · VGGSound test, 1 step | Acc (%): 94.96 -> 97.85 | Helped |
| add reflow (ablation) | Frieren · VGGSound test, 25 steps | IS: 12.42 -> 9.79 | Hurt |
| increase vector field estimator size from Small to Base (ablation) | Frieren · VGGSound test | IS: 12.16 -> 12.42 | Helped |
| increase vector field estimator size from Base to Large (ablation) | Frieren · VGGSound test | Acc (%): 97.22 -> 95.16 | Hurt |
| replace CAVP visual features with MAViL features (ablation) | Frieren · VGGSound test | Acc (%): 97.22 -> 90.17 | Hurt |
| replace CAVP visual features with MAViL features (ablation) | Frieren · VGGSound test | FD: 12.26 -> 12.08 | Helped |
| add logit-normal time re-weighting to the RFM objective (ablation) | Frieren · VGGSound test | IS: 12.20 -> 12.42 | Helped |
| add logit-normal time re-weighting to the RFM objective (ablation) | Frieren · VGGSound test | KID (10^-3): 2.12 -> 2.49 | Hurt |
| use CFG scale 4.5 rather than scales around 2–3 (ablation) | Frieren · VGGSound test | IS | Mixed |
| use BigVGAN instead of Griffin-Lim (ablation) | Frieren · VGGSound test | IS: 10.67 -> 12.42 | Helped |
Surprising findings
- Reflow slightly degrades 25-step quality (IS 9.79 vs 12.42 without reflow), which the authors attribute to limited steps in reflow data generation.
- Adding classifier guidance to Diff-Foley hurts quality at 5 steps, suggesting its alignment mechanism is fragile when steps are reduced.
Limitations
- Experiments only on the small-scale VGGSound dataset; not scaled to large data.
- Only supports fixed-length short video clips (8 seconds), not variable-length long videos.
- Reflow slightly reduces 25-step quality, and the cause is only speculated.
Experimental setup
Datasets: VGGSound AudioSet
Benchmarks: VGGSound
Metrics: FD IS KL FAD KID Acc MOS-Q MOS-A
Baselines: SpecVQGAN Im2Wav Diff-Foley LDM
Builds on: rectified flow flow matching Diff-Foley BigVGAN DPM-Solver
Closest papers by meaning
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.89)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.88)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.87)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.87)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.87)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.