Research map / Video / Video generation
MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMDisCo, a lightweight discriminator-based joint guidance module that cooperatively guides two frozen pre-trained single-modal diffusion models (audio and video) to generate aligned audio-video pairs.
Problem
Building audio-video joint generative models cheaply by integrating pre-trained single-modal diffusion models, while keeping per-modality fidelity and cross-modal alignment without model-specific architectural redesign.
Approach
Extends classifier guidance to joint generation: a density-ratio discriminator trained to separate real audio-video pairs from independently generated fake pairs yields the cross-modal score correction, and a DLSM-style denoising regularizer stabilizes its gradient so it acts as a noise estimator. Base models are treated as black boxes and are not updated.
Key results
- On Landscape (OOD), AudioLDM/AnimateDiff + MMDisCo improves FVD from 852 to 667 and FAD from 8.26 to 7.69, with IB-AV rising from 0.093 to 0.102.
- On VGGSound (OOD), Auffusion/VideoCrafter2 + MMDisCo reduces FVD from 831 to 704 and FAD from 5.32 to 4.79, with IB-AV rising from 0.192 to 0.201.
- On the in-domain MM-Diffusion setting, the guidance module (12.7M params vs 133M base) improves FVD 447→405 on Landscape and 513→450 on AIST++.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MMDisCo guidance | MM-Diffusion · Landscape and AIST++ (in-domain) | FVD: 447 -> 405; 513 -> 450 | Helped |
| add MMDisCo guidance | MM-Diffusion · Landscape and AIST++ (in-domain) | FAD: 5.78 -> 5.52; 2.31 -> 2.17 | Helped |
| add MMDisCo guidance | MM-Diffusion · Landscape and AIST++ (in-domain) | IB-AV: 0.156 -> 0.162; 0.0897 -> 0.0909 | Helped |
| add MMDisCo guidance | AudioLDM / AnimateDiff · Landscape (out-of-domain) | IB-TV: 0.308 -> 0.303 | Hurt |
| add MMDisCo guidance | AudioLDM / AnimateDiff · VGGSound (out-of-domain) | FVD: 739 -> 754 | Hurt |
| add MMDisCo guidance | AudioLDM / AnimateDiff · VGGSound (out-of-domain) | IB-AV: 0.121 -> 0.127 | Helped |
| combine L_disc and L_denoise rather than use either loss alone (ablation) | MMDisCo with AudioLDM / AnimateDiff · Landscape (out-of-domain) | IB-AV: 0.102 (L_disc); 0.088 (L_denoise); 0.102 (L_all) | Mixed |
| increase discriminator channel size from C=32 to C=128 (ablation) | MMDisCo with MM-Diffusion · Landscape (in-domain) | FVD: 423 -> 405 | Helped |
| increase discriminator channel size from C=128 to C=256 (ablation) | MMDisCo with MM-Diffusion · Landscape (in-domain) | IB-AV: 0.162 -> 0.160 | Hurt |
| increase ResBlocks per resolution from L=2 to L=4 at C=128 (ablation) | MMDisCo with MM-Diffusion · Landscape (in-domain) | FAD: 5.52 -> 5.43 | Helped |
| train the discriminator beyond 100 epochs (ablation) | MMDisCo with MM-Diffusion · Landscape (in-domain) | performance | No clear effect |
| replace MLP feature fusion with Frieren transformer (ablation) | MMDisCo with Auffusion / VideoCrafter2 · VGGSound (out-of-domain) | IB-AV: 0.201 -> 0.205 | Helped |
Surprising findings
- Using a discriminator trained only with the discriminator loss degrades single-modal fidelity; the DLSM-style denoising regularizer is needed to keep fidelity.
- On VGGSound, the joint guidance slightly worsens FVD (739→754) and IB-TV for the AudioLDM/AnimateDiff pair.
Limitations
- Output quality depends strongly on the pre-trained base models.
- Uses the most basic density-ratio estimator; more advanced ones are left for future work.
- Guidance scale tuning is needed in some settings (e.g., a scale of 8 was grid-searched for Auffusion/VideoCrafter2 video latents).
Experimental setup
Datasets: Landscape AIST++ VGGSound
Benchmarks: Landscape AIST++ VGGSound
Metrics: FVD FAD IB-AV IB-TV IB-TA
Baselines: MM-Diffusion CoDi AnimateDiff → SpecVQGAN AnimateDiff → DiffFoley VideoCrafter2 → DiffFoley VideoCrafter2 → SpecVQGAN
Builds on: classifier guidance discriminator guidance DLSM MM-Diffusion DDPM
Closest papers by meaning
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.90)
- A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.89)
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.89)
- MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation (ICLR 2025 · similarity 0.88)
- Animate and Sound an Image (CVPR 2025 · similarity 0.88)
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.