Research map / Video / Video generation
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly denoise audio and video with coupled U-Nets rather than use the single-modality video stream | MM-Diffusion on Landscape · Landscape | FVD: 238.33 -> 229.08 | Helped |
| Jointly denoise audio and video with coupled U-Nets rather than use the single-modality audio stream | MM-Diffusion on Landscape · Landscape | FAD: 13.6 -> 9.39 | Helped |
| Use complete DDPM sampling instead of DPM-Solver (ablation) | MM-Diffusion on Landscape · Landscape | FVD: 229.08 -> 117.20 | Helped |
| Use complete DDPM sampling instead of DPM-Solver (ablation) | MM-Diffusion on AIST++ · AIST++ | FVD: 176.55 -> 75.71 | Helped |
| Use adaptive cross-modal attention windows [1,4,8] instead of [1,1,1] (ablation) | MM-Diffusion · Landscape (80K training steps) | FVD: 374.18 -> 303.19 | Helped |
| Use adaptive cross-modal attention windows [1,4,8] instead of [8,8,8] (ablation) | MM-Diffusion · Landscape (80K training steps) | FAD: 9.50 -> 10.20 | Hurt |
| Use random-shift cross-modal attention (ablation) | MM-Diffusion · Landscape (100K iterations) | FAD: 9.94 -> 9.10 | Helped |
| Use random-shift cross-modal attention (ablation) | MM-Diffusion · Landscape (60K iterations) | FAD: 12.55 -> 13.29 | Hurt |
| Use random-shift cross-modal attention (ablation) | MM-Diffusion · Landscape (60K/80K/100K iterations) | FVD: 415.50/ 303.19/ 271.64 -> 440.78/ 306.42/ 267.58 | Mixed |
| Use gradient guidance rather than replacement-based conditioning | MM-Diffusion zero-shot audio-to-video · Audio-to-video conditional generation | semantic and temporal alignment | Helped |
Closest papers by meaning
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.93)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.90)
- A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.90)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.89)
- Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusion (AAAI 2025 · similarity 0.88)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.