Research map / Video / Video generation
MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MDSGen, a lightweight masked diffusion transformer for video-to-audio generation with a temporal-aware masking strategy and a learnable video-feature Reducer, reaching 97.9% alignment accuracy with a 5M-parameter model on VGGSound.
Problem
Video-to-audio (open-domain sound) generation is dominated by large U-Net diffusion models with hundreds of millions of parameters; they are costly and redundant video features (CAVP) hurt generation quality.
Approach
Replaces U-Net with a pure masked diffusion transformer (DiT + MDT-style masking) for video-conditioned sound, uses temporal-aware masking (TAM) instead of spatial masking, and compresses 32 CAVP frame features into a single 768-d condition vector via a learnable Reducer.
Key results
- MDSGen-T (5M) reaches 97.91% alignment accuracy on VGGSound vs 93.92% for Diff-Foley (860M), with 0.01s inference vs 0.36s.
- MDSGen-B (131M) reaches 98.55% alignment accuracy and FID 11.19 at 500k steps, with IS 52.77.
- On Flickr-SoundNet, MDSGen-B achieves the best CIoU (82.01 vs 81.78 for FoleyCrafter).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use temporal-aware masking instead of random spatial-aware masking (ablation) | MDSGen · VGGSound test | IS: 48.66 -> 52.77 | Helped |
| Add temporal masking instead of using no masking (ablation) | MDSGen · VGGSound test | alignment accuracy: 97.12 -> 98.55 | Helped |
| Reduce video features from 32 × 768 to 1 × 768 (ablation) | MDSGen-B · VGGSound test | alignment accuracy: 96.18 -> 98.55 | Helped |
| Use learnable-weight pooling instead of attention pooling (ablation) | MDSGen-B · VGGSound test, 300k steps | IS: 39.0696 -> 46.3301 | Helped |
| Use four decoder layers instead of two (ablation) | MDSGen-B · VGGSound test, 300k steps | alignment accuracy: 0.9823 -> 0.9858 | Helped |
| Add classifier-free guidance (ablation) | MDSGen-B · VGGSound test | alignment accuracy: 84.1 -> 98.6 | Helped |
| Add classifier guidance to classifier-free guidance (ablation) | MDSGen-B · VGGSound test | alignment accuracy: 98.6 -> 98.8 | Helped |
| Increase temporal masking ratio from 30% to 50% (ablation) | MDSGen-B · VGGSound test | FID: 11.19 -> 12.39 | Hurt |
| Increase temporal masking ratio from 30% to 70% (ablation) | MDSGen-B · VGGSound test | FID: 11.19 -> 12.85 | Hurt |
| Select the G channel instead of the R channel (ablation) | MDSGen-B with ImageVAE · VGGSound test | FID: 11.40 -> 11.19 | Helped |
| Replace ImageVAE with AudioLDM VAE (ablation) | MDSGen-B · VGGSound test | FAD: 2.36 -> 1.34 | Helped |
| Replace ImageVAE with AudioLDM VAE (ablation) | MDSGen-B · VGGSound test | alignment accuracy: 98.58 -> 96.52 | Hurt |
Surprising findings
- Image-trained Stable Diffusion VAE outputs differ across RGB channels even though the input is replicated grayscale; the G channel works best, unlike Diff-Foley's R channel.
- Larger L model (461M) underperforms B, suggesting overfitting at larger scale on VGGSound.
Limitations
- Requires multiple sampling steps, as with other diffusion models.
- Fixed 8.2-second video length constraint.
- VGGSound size may limit the approach; alignment metric depends on a classifier trained by Diff-Foley's authors.
Experimental setup
Datasets: VGGSound Flickr-SoundNet
Benchmarks: VGGSound test Flickr-SoundNet test
Metrics: FID IS KL FAD alignment accuracy CIoU AUC inference time parameter count memory usage
Baselines: SpecVQGAN Im2Wav Diff-Foley See and Hear FoleyCrafter
Builds on: Diffusion Transformer (DiT) Masked Diffusion Transformer (MDT) Diff-Foley CAVP DPM-Solver
Closest papers by meaning
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation (NeurIPS 2025 · similarity 0.89)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.88)
- Fast Timing-Conditioned Latent Audio Diffusion (ICML 2024 · similarity 0.88)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.88)
- A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.87)
- IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.