Research map / Image generation / Autoregressive and latent generation
Matryoshka Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Matryoshka Diffusion (MDM), an end-to-end diffusion model that jointly denoises multiple resolutions with a NestedUNet and progressive low-to-high resolution training, enabling single pixel-space models up to 1024x1024 trained on CC12M.
Problem
High-resolution diffusion synthesis is computationally and optimization-wise hard; existing approaches rely on cascaded pixel-space models or latent autoencoders, which complicate training and bound quality.
Approach
Denoise multiple resolutions jointly in an extended latent space with a NestedUNet whose low-resolution features and parameters are shared by higher resolutions, combined with a multi-resolution loss and progressive training from low to high resolution.
Key results
- On ImageNet 256x256, MDM with CFG reaches FID 3.51, comparable to LDM-4* (3.60) and CDM (4.88).
- Zero-shot COCO FID-30K at 256x256 is 13.43 with CFG (18.35 without), versus Dalle-2* 10.39 and Imagen* 7.27.
- Progressive training with more low-resolution steps improves high-resolution FID convergence; MDM converges faster than Simple DM, Cascaded DM and Latent DM baselines under controlled settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-resolution diffusion loss instead of a single-resolution loss (ablation) | MDM · ImageNet 256×256 | FID | Helped |
| use multi-resolution diffusion loss instead of a single-resolution loss (ablation) | MDM · CC12M 256×256 | FID | Helped |
| add progressive training from lower to higher resolution (ablation) | MDM · ImageNet 256×256 | FID | Helped |
| add progressive training from lower to higher resolution (ablation) | MDM · CC12M 256×256 | FID | Helped |
| increase the number of low-resolution training iterations before progressing (ablation) | MDM · ImageNet 256×256 | FID | Helped |
| increase the number of nested resolution levels from two to three (ablation) | MDM · CC12M | CLIP score | Helped |
| use NestedUNet instead of standard UNet as a standalone architecture (ablation) | single-resolution diffusion model · ImageNet 256×256 | FID | No clear effect |
| apply classifier-free guidance to MDM | MDM · ImageNet 256×256 | FID-50K: 8.18 -> 3.51 | Helped |
| apply classifier-free guidance to MDM | MDM · MS-COCO 256×256 | FID-30K: 18.35 -> 13.43 | Helped |
| replace MDM with Cascaded DM | MDM · ImageNet 256×256 | FID | Hurt |
| replace MDM with Latent DM | MDM · ImageNet 256×256 | FID | Hurt |
| increase classifier-free guidance weight (ablation) | MDM · MS-COCO 256×256 | FID | Mixed |
Surprising findings
- Cascaded DM underperforms MDM despite having more combined parameters and twice as many inference steps.
- Nested UNet alone without the multi-resolution loss offers no advantage over a standard UNet in speed or FID.
Limitations
- Results are not state-of-the-art and trained on relatively small datasets (CC12M, WebVid-10M) with limited model sizes.
- Architecture and hyperparameters were not optimized, e.g., for ImageNet.
- Video experiments are preliminary, at 16 frames of 256x256 and one clip per video.
Experimental setup
Datasets: ImageNet CC12M WebVid-10M MS-COCO
Benchmarks: ImageNet 256x256 CC12M 256x256 MS-COCO 256x256
Metrics: FID CLIP score
Baselines: ADM CDM LDM-4 LDM-8 Dalle-2 IMAGEN Simple DM Cascaded DM Latent DM
Builds on: UNet Imagen f-DM progressive growing of GANs v-prediction
Closest papers by meaning
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.89)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.89)
- Hierarchical Patch Diffusion Models for High-Resolution Video Generation (CVPR 2024 · similarity 0.88)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.88)
- Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (ICLR 2025 · similarity 0.88)
- Unleashing the Power of One-Step Diffusion based Image Super-Resolution via a Large-Scale Diffusion Discriminator (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.