Research map / Video / Video generation
Native Audio-Visual Alignment for Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NAVA, a 6.3B joint audio-video generator that aligns audio and video in a dedicated interaction space before fusing context via cross-attention, plus timbre-in-context conditioning for segment-level speech timbre control.
Problem
Open-source joint audio-video generation either uses dual-tower designs with late posterior alignment, which weakens fine-grained audio-video co-evolution, or fully unified tri-modal attention, which entangles semantic conditioning with low-level synchronization.
Approach
Separate audio-video synchronization from context conditioning: early modality-aware layers do joint audio-video self-attention, context enters only via cross-attention, later layers share parameters (Align-then-Fuse MMDiT). Reference timbre tokens are placed inside the speech-span markers in the prompt so each timbre binds to its span; a condition-factorized CFG separates text, alignment and timb
Key results
- On Verse-Bench, NAVA (6.3B) reaches Sync-C 7.791 and Sync-D 7.566, the best among compared audio-video models, with the fewest parameters.
- On Seed-TTS EN, NAVA gets WER 4.20 and speaker similarity 66.7, versus DreamID-Omni's 31.76 WER and 35.7 similarity.
- In the user study, NAVA wins 67.5%, 60.0%, and 80.0% of T2AV overall-quality comparisons against Ovi-1.1, LTX-2.3, and daVinci.
Surprising findings
- Alignment CFG raises Sync-C from 6.170 to 7.791 on Verse-Bench, a large effect from guidance alone.
- Timbre CFG improves speaker similarity but worsens WER from 3.78 to 4.20.
Limitations
- Struggles with long-tail and compositional audio events such as rare animal sounds, music, singing, and complex scene mixtures.
- Depends on large, carefully curated audio-visual data.
- MoVA excluded from T2AV user study because it cannot take text-only input.
Experimental setup
Datasets: Koala-36M TED-style speech videos movie/TV footage Verse-Bench Seed-TTS
Benchmarks: Verse-Bench Seed-TTS
Metrics: Sync-C Sync-D IB-Score video quality WER PQ FD speaker similarity win/tie/lose user preference
Baselines: Ovi-1.1 MoVA LTX-2.3 daVinci-MagiHuman DreamID-Omni CosyVoice CosyVoice2 Qwen2.5-Omni
Builds on: Wan2.2 LTX-2 MMDiT
Closest papers by meaning
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.91)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.90)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.90)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.89)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.89)
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.