Research map / Video / Video generation
Animate and Sound an Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JointDiT, which fuses pretrained SVD (video) and AudioLDM2 (audio) diffusion models into a joint DiT with a shared full-attention block and a joint classifier-free guidance, for image-to-sounding-video generation.
Problem
Generating a sounding video from a single static image, with video and audio synchronized and semantically consistent, is underexplored; existing pipelines suffer from error propagation or coarse cross-modal interaction.
Approach
Decompose two pretrained unimodal diffusion models into input, expert, and output layers and join their expert layers with a full-attention Joint Block using Perceiver-style modality-specific QK/V projections; add a joint CFG that contrasts the full model against modality-isolated (attention-masked) and "bad" versions.
Key results
- On AVSync15, JointDiT reports FVD 326 and KVD 15.2 versus 444/37.7 for the best SVD-based pipelines, and IB-VA 87.7 (second best 87.0).
- On Landscape, JointDiT achieves the best results on every reported metric; on GreatestHits it leads in FVD (173) and FAD (1.08).
- In a 10-annotator user study on AVSync15, JointDiT scores highest on all five dimensions (e.g., Overall 1.52 vs 1.28 for the next best).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JointCFG* instead of vanilla CFG (ablation) | JointDiT · AVSync15 test subset | FVD: 662.8 -> 653.7 | Helped |
| Use JointCFG instead of vanilla CFG (ablation) | JointDiT · AVSync15 test subset | FVD: 662.8 -> 661.6 | Helped |
| Use JointCFG* instead of vanilla CFG (ablation) | JointDiT · AVSync15 test subset | AV-Align: 1.266 -> 1.288 | Mixed |
| Add the Joint Block (ablation) | JointDiT · AVSync15 test subset | FVD: 819.5 -> 653.7 | Helped |
| Add the Joint Block (ablation) | JointDiT · AVSync15 test subset | FAD: 30.1 -> 27.6 | Helped |
| Add the Joint Block (ablation) | JointDiT · AVSync15 test subset | IB-VA: 36.2 -> 37.4 | Helped |
| Replace standard self-attention with Perceiver Joint Attention (ablation) | JointDiT · AVSync15 test subset | FVD: 921.2 -> 653.7 | Helped |
| Replace standard self-attention with Perceiver Joint Attention (ablation) | JointDiT · AVSync15 test subset | FAD: 29.3 -> 27.6 | Helped |
| Replace standard self-attention with Perceiver Joint Attention (ablation) | JointDiT · AVSync15 test subset | IB-VA: 34.4 -> 37.4 | Helped |
| Replace independent or sequential generation with JointDiT | AVSync15 · AVSync15 | FVD: 444 -> 326 | Helped |
| Replace independent or sequential generation with JointDiT | AVSync15 · AVSync15 | AV-Align: 1.285 -> 1.296 | Mixed |
| Replace baseline methods with JointDiT | GreatestHits · GreatestHits | FVD: 287 -> 173 | Helped |
Surprising findings
- Automatic metrics did not always agree with human ratings; e.g., AudioLDM-v+AVSyncD ranked higher than JointDiT on AV-Align, while humans ranked it second for synchronization.
- Naive full attention with shared channel projections fails to model cross-modal interactions, motivating the Perceiver Joint Attention design.
Limitations
- Automatic evaluation metrics only partially align with human judgments on synchronization.
- Current evaluation is limited to image-conditioned generation on three datasets; text-conditioned and other cross-modal tasks are left for future work.
- JointCFG* slightly degrades AV-Align compared with vanilla CFG.
Experimental setup
Datasets: AVSync15 Landscape GreatestHits
Benchmarks: AVSync15 Landscape GreatestHits
Metrics: FVD KVD FAD KL IB-VA AV-Align IB-IV IB-IA Motion Score user study scores
Baselines: SVD + AudioLDM-v SVD + IM2Wav SVD + Diff-Foley SVD + FoleyCrafter SVD + TiVA SVD + SeeingHearing AVSyncD CoDi
Builds on: Stable Video Diffusion AudioLDM2 classifier-free guidance Autoguidance Perceiver attention
Closest papers by meaning
- Read, Watch and Scream! Sound Generation from Text and Video (AAAI 2025 · similarity 0.88)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.88)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.88)
- A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.87)
- AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers (CVPR 2025 · similarity 0.87)
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.