Research map / Image generation / Autoregressive and latent generation
Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Kaleido, which adds an autoregressive latent prior (text, bounding boxes, object blobs, or visual tokens) that conditions a diffusion model to keep sample diversity high under large classifier-free guidance.
Problem
Diffusion models lose sample diversity at high classifier-free guidance weights, so the same caption yields visually similar images.
Approach
Model discrete abstract latents (detailed captions, grounded boxes, tilted-ellipse blobs, or SEED visual tokens) autoregressively from the condition, then sample these as "mode selection" before diffusion so that CFG sharpens only the image-variation distribution, and train the diffusion model jointly with the AR decoder via an ELBO-style objective.
Key results
- Kaleido improves Recall relative to MDM across CFG scales while keeping FID competitive, with the baseline's FID rising and Recall dropping markedly at high CFG (exact numbers not given in the provided text).
- Generated latents (text, bbox, blob, voken) are human-readable and images align closely with them, and latent editing yields predictable changes in the output.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add autoregressive latent priors to MDM | Kaleido Diffusion vs. MDM · ImageNet and CC12M, 256×256 | Recall | Helped |
| Add autoregressive latent priors to MDM | Kaleido Diffusion vs. MDM · ImageNet and CC12M, 256×256 | FID | Helped |
| Add autoregressive latent priors to MDM | Kaleido Diffusion vs. MDM · ImageNet and CC12M, high CFG | sample diversity | Helped |
Limitations
- Training is more complex and costly, roughly 3x slower than MDM, with higher memory usage from the T5 decoder.
- Finding the optimal latent type for a desired diversity level requires extensive experimentation.
- Only four latent types are explored; other modes such as depth or semantic maps are not covered.
Experimental setup
Datasets: ImageNet CC12M
Benchmarks: ImageNet 256x256 CC12M 256x256
Metrics: FID Recall
Baselines: Matryoshka Diffusion Models (MDM)
Builds on: Matryoshka Diffusion Models classifier-free guidance T5 BlobGen
Closest papers by meaning
- Nested Diffusion Models Using Hierarchical Latent Priors (CVPR 2025 · similarity 0.90)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.90)
- Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.89)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.89)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.