Research map / Image generation / Fast diffusion sampling
A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture of Noise Levels (MoNL), training a transformer-based audiovisual latent diffusion model (AVDiT) with per-modality and per-time-segment diffusion timestep vectors so one model handles joint, cross-modal, and interpolation/continuation generation.
Problem
Diffusion models for audiovisual sequences typically need a separate model per conditional task; a single task-agnostic model that learns arbitrary conditional distributions over audio and video is needed, with temporally consistent outputs.
Approach
Replace the scalar diffusion timestep with a vector of timesteps over modalities and time-segments, randomly sampling among vanilla, per-modality, per-time-segment, and per-time-segment-and-modality schemes so that setting timesteps to 0 for conditioning parts yields arbitrary conditional inference without retraining or inference-time modifications.
Key results
- On Monologues, MoNL achieves the best average FAD/FVD across tasks (average FAD 4.9, FVD 34.6) versus Vanilla (7.4 / 121.1) and task-specific conditional models (7.8 / 35.3).
- On AIST++ and Landscape, MoNL beats MM-Diffusion on A2V FVD (38.04 vs 184.45 on AIST++; 86.79 vs 238.33 on Landscape) and on V2A FAD (1.11 vs 13.30 on AIST++).
- In user studies vs MM-Diffusion, raters preferred MoNL on AV align (0.61-0.77), AV quality (0.49-0.72) and person consistency (0.60-0.93) across tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoNL instead of task-specific conditional models | AVDiT · Monologues, AV-inpaint | FVD: 15.9 -> 11.8 | Helped |
| Use MoNL instead of task-specific conditional models | AVDiT · Monologues, AV-continue | FVD: 12.1 -> 8.8 | Helped |
| Use MoNL instead of the Per modality design | AVDiT · Monologues, A2V | FVD: 34.1 -> 40.2 | Hurt |
| Use MoNL instead of the Per modality design | AVDiT · Monologues, V2A | FAD: 4.7 -> 5.3 | Hurt |
| Use MoNL instead of MM-Diffusion | AVDiT · AIST++, A2V | FVD: 184.45 -> 38.04 | Helped |
| Use MoNL instead of MM-Diffusion | AVDiT · Landscape, A2V | FVD: 238.33 -> 86.79 | Helped |
| Use MoNL instead of MM-Diffusion | AVDiT · AIST++, V2A | FAD: 13.30 -> 1.11 | Helped |
| Use MoNL instead of MM-Diffusion | AVDiT · Landscape, V2A | FAD: 13.60 -> 0.78 | Helped |
| Add Vanilla to the timestep mixture (ablation) | AVDiT trained with Pt/Pm/Ptm · Monologues, Joint | FVD: 90.1 -> 77.6 | Helped |
| Use Per time-segment noise design alone instead of MoNL (ablation) | AVDiT · Monologues, AV-continue | FVD: 8.8 -> 72.3 | Hurt |
| Use Per time-segment and Per modality noise design alone instead of MoNL (ablation) | AVDiT · Monologues, AV-continue | FVD: 8.8 -> 61.2 | Hurt |
| Exclude Vanilla from the timestep mixture (ablation) | AVDiT · Monologues, Joint | FVD: 77.6 -> 90.1 | Hurt |
Surprising findings
- Per-modality timestep sampling beats MoNL on cross-modal A2V and V2A tasks in FVD/FAD, while MoNL excels on interpolation/continuation.
- Vanilla joint training achieves the best unconditional joint generation FAD/FVD but performs poorly on all conditional tasks.
Limitations
- Visual and speech quality remain limited, with blurry faces and reconstruction artifacts from pretrained autoencoders.
- Mixture scheme is a simple uniform random choice among four timestep designs; weighted mixing is unexplored.
- Experiments focus on human-centric talking-head data; potential stereotype and bias concerns for A2V/V2A.
Experimental setup
Datasets: Monologues AIST++ Landscape
Benchmarks: Monologues AIST++ Landscape
Metrics: FAD FVD KVD AV-align user preference (Likert)
Baselines: MM-Diffusion Vanilla diffusion Per-modality (UniDiffuser-style) Task-specific conditional models
Builds on: UniDiffuser W.A.L.T MAGVIT-v2 SoundStream MM-Diffusion
Closest papers by meaning
- Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces (ICML 2025 · similarity 0.89)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.89)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.89)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
- Diffusion-TS: Interpretable Diffusion for General Time Series Generation (ICLR 2024 · similarity 0.88)
- Diffusion Models for Multi-Task Generative Modeling (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.