Research map / Video / Video generation
Audio-Sync Video Generation with Multi-Stream Temporal Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTV, an audio-sync video generation framework that demixes audio into speech, effects, and music tracks and controls a CogVideoX-based diffusion transformer via a multi-stream temporal ControlNet, trained with a multi-stage schedule on the new DEMIX dataset.
Problem
Generating cinematic videos with precise audio-visual synchronization (lip motion, event timing, visual mood) from diverse and complex audio, where existing methods map composite audio to scenes ambiguously and align timing poorly.
Approach
Explicitly demix audio into speech, effects, and music tracks; route speech and effects through an interval stream that injects per-time-interval features, and music through a holistic stream that applies global style modulation, trained stage-by-stage on five overlapping data subsets.
Key results
- MTV reaches FVD 626.06 versus 795.88 to 879.77 for baselines on the comparison set.
- MTV improves Audio-C to 26.22% (baselines 5.43% to 7.30%) and Sync-C to 3.17 (baselines 1.45 to 1.55).
- Removing the separate-extraction design raises FVD to 667.81; freezing the DiT backbone (W/o TB) lowers Sync-C to 2.31.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MTV instead of the strongest baseline | Audio-sync video generation · Audio-sync video generation test set | FVD: 795.88 -> 626.06 | Helped |
| Use MTV instead of the strongest baseline | Audio-sync video generation · Audio-sync video generation test set | Audio-C (%): 7.30 -> 26.22 | Helped |
| Use MTV instead of the strongest baseline | Audio-sync video generation · Audio-sync video generation test set | Sync-C: 1.55 -> 3.17 | Helped |
| Add separate extraction so music features can shape overall visual mood (ablation) | MTV · Audio-sync video generation test set | FVD: 626.06 -> 667.81 | Helped |
| Add separate track injection instead of shared cross-attention (ablation) | MTV · Audio-sync video generation test set | Audio-C (%): 19.64 -> 26.22 | Helped |
| Train the DiT backbone together with MST-ControlNet (ablation) | MTV · Audio-sync video generation test set | Sync-C: 2.31 -> 3.17 | Helped |
Surprising findings
- AV-Align scores rank real videos lowest, so the authors argue the metric is unsuitable for cinematic videos and report it only in the supplement.
Limitations
- Limited by the categories provided by upstream audio demixing tools.
- Comparison set is small (50 videos) and baselines were finetuned or run in their original configurations.
- Inference is slow at 280 s per 49-frame clip.
Experimental setup
Datasets: CelebV-HQ MovieBench Condensed Movies Short-Films 20K YouTube film videos
Benchmarks: DEMIX test set (1K clips, 50 sampled for comparison)
Metrics: FVD Temp-C Text-C Audio-C Sync-C Sync-D AV-Align
Baselines: MM-Diffusion TempoTokens Xing et al.
Builds on: CogVideoX ControlNet wav2vec MM-Diffusion
Closest papers by meaning
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.89)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.88)
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.88)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.88)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.87)
- Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decomposition (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.