Research map / Image generation / Autoregressive and latent generation
Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free, optimization-based diffusion latent aligner that uses pretrained ImageBind embeddings to guide off-the-shelf AudioLDM and AnimateDiff for joint, V2A, A2V and I2A generation.
Problem
Open-domain visual-audio generation is limited because video and audio are generated separately, and existing V2A, A2V and joint generation methods are domain-limited or perform poorly.
Approach
Bridge pretrained single-modality diffusion models through a shared ImageBind embedding space: at each denoising step, predict the clean latent, compute a cosine-distance loss to the condition embedding, and take gradient steps on the noisy latent (plus a learned text prompt embedding for guided prompt tuning), with no extra training.
Key results
- On V2A, the method reaches KL 2.619 vs 3.290 for SpecVQGAN on 3k VGGSound samples, with higher ISc (5.831 vs 5.108).
- On A2V, FVD 402.385 and KVD 34.764 with AV-align 0.522, versus TempoTokens FVD 1866.285 and AV-align 0.423.
- On Landscape joint generation, adding the aligner to MM-Diffusion lowers FAD from 7.752 to 6.463 while FVD rises slightly (1141.009 to 1174.856).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add diffusion latent alignment guidance | AudioLDM for V2A · VGGSound (3k samples) | KL: 3.203 -> 2.619 | Helped |
| add diffusion latent alignment guidance | AudioLDM for V2A · VGGSound (3k samples) | FD: 40.457 -> 32.920 | Helped |
| add diffusion latent alignment guidance | AudioLDM for I2A · VGGSound (3k samples) | FAD: 7.364 -> 6.869 | Helped |
| add diffusion latent alignment guidance | AudioLDM for I2A · VGGSound (3k samples) | FD: 33.049 -> 20.958 | Helped |
| add diffusion latent alignment guidance | AnimateDiff for A2V · VGGSound (3k samples) | FVD: 417.398 -> 402.385 | Helped |
| add diffusion latent alignment guidance | MM-Diffusion · Landscape, Joint VA Generation | FAD: 7.752 -> 6.463 | Helped |
| add diffusion latent alignment guidance | MM-Diffusion · Landscape, Joint VA Generation | FVD: 1141.009 -> 1174.856 | Hurt |
| add diffusion latent alignment guidance | AudioLDM + AnimateDiff · Open-domain, Joint VA Generation | AT-alignbind: 0.081 -> 0.138 | Helped |
| add diffusion latent alignment guidance | AudioLDM + AnimateDiff · Open-domain, Joint VA Generation | AV-align: 0.226 -> 0.283 | Helped |
Surprising findings
- On the I2A task, the training-free method's KL (2.691) is worse than Im2Wav's (2.612), so it does not win on every metric.
- Guidance on the audio-to-video direction alone had a negligible effect, which motivated the guided prompt tuning component.
Limitations
- Relies on the quality of off-the-shelf generators (AudioLDM, AnimateDiff) and on ImageBind's embedding space.
- Audio-to-video guidance alone has negligible effect and video temporal consistency can degrade without prompt tuning.
- Evaluated on relatively small samples (3k per task, 200 for Landscape joint generation) with no human evaluation.
Experimental setup
Datasets: VGGSound Landscape
Benchmarks: VGGSound Landscape
Metrics: KL ISc FD FAD FVD KVD AV-align AV-alignbind VT-alignbind AT-alignbind
Baselines: SpecVQGAN Im2Wav TempoTokens MM-Diffusion AudioLDM-based vanilla pipeline AnimateDiff-based vanilla pipeline
Builds on: classifier guidance ImageBind AudioLDM AnimateDiff MM-Diffusion
Closest papers by meaning
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.90)
- A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.89)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.89)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.89)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.89)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.