Research map / Image generation / Autoregressive and latent generation

Nested Diffusion Models Using Hierarchical Latent Priors

CVPR 2025 · Image generation › Autoregressive and latent generation · approach: Latent diffusion with learned tokenizers · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes nested diffusion models: a stack of U-ViT diffusion models that generate hierarchical semantic latents (from frozen MoCo-v3/CLIP patch features, reduced via SVD and noised) before the image, improving unconditional and conditional ImageNet and COCO generation FID.

Problem

Diffusion models generating images in a single latent space struggle with complex scenes, and hierarchical VAEs suffer from training instability and collapse; a principled multi-level generative hierarchy with semantic structure is needed.

Approach

Anchor a cascade of diffusion models to a hierarchy of frozen pre-trained visual encoder features computed on progressively smaller image patches, with capacity controlled by SVD channel reduction and Gaussian noise injection, so each level is conditioned on the more abstract semantic latents above it.

Key results

• On ImageNet-1K, the 5-level unconditional model reduces FID without CFG from 55.41 (single-level U-ViT) to 11.05, and the class-conditional model from 31.13 to 9.87.
• With CFG, the 5-level model achieves FID 3.97 conditional and 5.03 unconditional on ImageNet-1K, versus 13.75 for the single-level baseline.
• On COCO-2014 without CFG, the 3-level model reaches FID 6.97 vs 14.98 for U-ViT, and the 2-level model with CFG reaches 4.72, beating the 7B CM3Leon (4.88).

Measured effects reported in the paper

Surprising findings

• The unconditional 5-level nested model surpasses the class-conditional single-level baseline on ImageNet-1K.
• Adding Gaussian noise to the hierarchical latents is essential; without it the top-level conditioning is bypassed and the generator behaves like an autoencoder.

Limitations

• Requires per-level noise hyperparameters (sigma_l) and CFG weight tuning via a greedy local search.
• Text-conditional results are on COCO only with a single CFG setting that the authors note may be suboptimal for L=3.

Experimental setup

Datasets: ImageNet-1K COCO-2014

Benchmarks: ImageNet-1K 256x256 COCO-2014

Metrics: FID KNN accuracy

Baselines: U-ViT DiT-XL/2 REPA GLIDE DALL-E 2 Imagen Re-Imagen CM3Leon-7B Parti-20B VQ-Diffusion Frido hierarchical VAE

Builds on: U-ViT denoising diffusion probabilistic models MoCo-v3 CLIP classifier-free guidance

Closest papers by meaning

• Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.92)
• Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling (NeurIPS 2024 · similarity 0.90)
• Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.90)
• Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.90)
• Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.90)
• Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.90)