Research map / Video / Video generation
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the text-to-video model ModelScope to audio conditioning by training a lightweight AudioMapper (over BEATs features) that maps expanding-context audio tokens to pseudo-text tokens, and introduces the AV-Align peak-alignment metric.
Problem
Generating diverse, realistic videos from natural audio that are aligned both globally (semantic class) and temporally (segment-level synchronization) with the input sound.
Approach
A lightweight adaptor maps BEATs audio embeddings into the text-token space of a frozen latent text-to-video diffusion model, using multi-scale expanding context windows and attentive pooling to produce per-frame temporal audio tokens; it also enables joint text+audio conditioning.
Key results
- On AudioSet-Drums, the method reaches AV-Align 0.61 vs 0.28 for TATS, with FVD 299 vs 303.
- On Landscape, FVD 784 and AV-Align 0.54 vs 0.41 for MM-Diffusion (FVD 922).
- On VGGSound, text-only ModelScope gives better FVD (801) and IS (15.55) than the audio-conditioned model (FVD 923, IS 11.04), but the audio model has higher AV-Align (0.35 vs 0.27).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use audio-conditioned generation instead of ModelScope Text2Vid | Ours · VGGSound | AV-Align: 0.27 -> 0.35 | Helped |
| Use audio-conditioned generation instead of ModelScope Random | Ours · VGGSound | AV-Align: 0.26 -> 0.35 | Helped |
| Use audio-conditioned generation instead of TATS | Ours · AudioSet-Drums | AV-Align: 0.28 -> 0.61 | Helped |
| Use audio-conditioned generation instead of MM-Diffusion | Ours · Landscape | FVD: 922 -> 784 | Helped |
| Combine text and audio conditioning instead of audio-only conditioning | Ours · VGGSound | AV-Align: 0.35 -> 0.36 | Helped |
| Combine text and audio conditioning instead of audio-only conditioning | Ours · VGGSound | FVD: 923 -> 859 | Helped |
| Use text-only conditioning instead of audio-only conditioning | Ours · VGGSound | AV-Align: 0.35 -> 0.27 | Hurt |
| Average all audio components into a single conditioning vector rather than using temporal windows (ablation) | Ours · VGGSound | AV-Align: 0.35 -> 0.29 | Hurt |
| Use one context-window resolution rather than five (ablation) | Ours · VGGSound | AV-Align: 0.35 -> 0.36 | Helped |
| Use two context-window resolutions rather than five (ablation) | Ours · VGGSound | AV-Align: 0.35 -> 0.35 | No clear effect |
| Use three context-window resolutions rather than five (ablation) | Ours · VGGSound | AV-Align: 0.35 -> 0.34 | Hurt |
| Use four context-window resolutions rather than five (ablation) | Ours · VGGSound | AV-Align: 0.35 -> 0.35 | No clear effect |
Surprising findings
- Text-only ModelScope conditioning yields better video quality than audio conditioning on VGGSound, while audio conditioning gives substantially better alignment.
Limitations
- Generated videos are short (24 frames) due to hardware limits.
- Audio-video mismatch in source data (e.g., off-screen sounds) can make alignment ill-posed.
- Mapping between audio and text token latent spaces may lose relevant audio information.
Experimental setup
Datasets: VGGSound AudioSet-Drums Landscape
Benchmarks: VGGSound AudioSet-Drums Landscape
Metrics: FVD CLIPSIM IS AV-Align MOS (semantic alignment, temporal alignment, quality)
Baselines: TATS MM-Diffusion ModelScope Text2Vid ModelScope Random
Builds on: ModelScope BEATs TATS MM-Diffusion AudioToken
Closest papers by meaning
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.91)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.91)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.90)
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.90)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.89)
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.