Research map / Video / Video generation
Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGAudio, a flow-based SiT Transformer for video-to-audio generation trained with a model-guidance objective (replacing CFG training) plus a dual-role CAVP audio-visual encoder used for conditioning and representation alignment, reaching FAD 0.40 on VGGSound.
Problem
Video-to-audio generation with diffusion/flow models typically relies on classifier-free guidance, whose random condition dropping may dilute model capacity and cause mismatched sampling behavior; open-domain V2A also needs high fidelity and data efficiency.
Approach
Adapts model guidance (MG), originally for class-conditional ImageNet, to continuous video conditions: the training target is self-distilled from the model's own conditional/unconditional predictions (EMA, stop-gradient), and a CAVP audio encoder is aligned with intermediate denoising features (REPA-style) while the CAVP video encoder provides AdaLN conditioning.
Key results
- MGAudio (131M) reaches FAD 0.40 and FD 6.16 on VGGSound with CFG=1.45, the best among compared methods, versus 0.71 for MMAudio.
- On UnAV-100 (zero-shot from VGGSound training), MGAudio achieves FAD 0.54 and FD 5.40, the best across listed baselines.
- Training on only 10% of VGGSound gives FAD 0.73 and FD 8.79 at 300k steps, outperforming full-data models on several metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with model guidance and use CFG = 1.45 at inference rather than the strongest listed CFG baseline | MGAudio · VGGSound test | FAD: 0.71 -> 0.40 | Helped |
| use MGAudio rather than MMAudio | VGGSound-trained models · UnAV-100 test | FAD: 0.93 -> 0.54 | Helped |
| add CFG at inference to MG-trained model (ablation) | MGAudio · VGGSound test | FAD: 0.80 -> 0.40 | Helped |
| add CFG at inference to MG without dual alignment (ablation) | SiT-based MG model · VGGSound test | FAD: 2.13 -> 1.19 | Helped |
| add dual alignment (ablation) | joint-guidance MGAudio · VGGSound test | FAD: 1.19 -> 1.14 | Helped |
| train on 10% rather than 100% of VGGSound (ablation) | MGAudio · VGGSound test | FAD: 0.81 -> 0.73 | Helped |
| remove AMG loss (ablation) | MGAudio trained on 10% of VGGSound · VGGSound test | FAD: 0.82 -> 32.84 | Hurt |
| use CAVP rather than DINOv2 as alignment encoder (ablation) | MGAudio · VGGSound test | FAD: 5.52 -> 1.14 | Helped |
| use CAVP rather than CLAP as alignment encoder (ablation) | MGAudio · VGGSound test | FAD: 1.35 -> 1.14 | Helped |
| increase model size from B/2 (131M) to XL/2 (680M) (ablation) | MGAudio · VGGSound test | FAD: 1.14 -> 0.90 | Helped |
| set alignment weight λ=0.5 instead of λ=0.25 (ablation) | MGAudio · VGGSound test | FAD: 0.74 -> 0.71 | Helped |
| increase alignment weight from λ=0.5 to λ=1.0 (ablation) | MGAudio · VGGSound test | FD: 12.07 -> 11.64 | Helped |
Surprising findings
- Model guidance alone underperforms SiT with CFG; the gains require combining MG training with CFG at inference, unlike the ImageNet setting where MG alone suffices.
- Using DINOv2 for alignment with MG yields poor results (FAD 5.52, alignment accuracy 21.66%), while CAVP audio features work best.
Limitations
- Weaker on human vocalizations, speech, and singing, which need phonetic awareness.
- Inference is slow due to VAE reliance and iterative sampling.
- Coarse global video conditioning (MDSGen-style) may limit temporal precision.
Experimental setup
Datasets: VGGSound UnAV-100
Benchmarks: VGGSound UnAV-100
Metrics: FAD FD IS KL Align. Acc. IB-Score DeSync
Baselines: Diff-Foley See and Hear FRIEREN MDSGen MMAudio SiT
Builds on: SiT Model Guidance (MG) REPA Diff-Foley MDSGen
Closest papers by meaning
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.90)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.89)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.89)
- MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation (ICLR 2025 · similarity 0.89)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.89)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.