Research map / Image generation / Autoregressive and latent generation
Latent diffusion with learned tokenizers
Apply diffusion and masked modeling in learned latent spaces for images and language.
114 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Autoregressive and latent generation, keyword cluster “latent space · modeling · vae”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
31 papers in 2023–24 and 74 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 6.5% to 8.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Uncorrelation and Evenness: a New Diversity-Promoting Regularizer (ICML 2017 · 18 citations)
- 2018 · Nonlocal Neural Networks, Nonlocal Diffusion and Nonlocal Modeling (NeurIPS 2018 · 29 citations)
- 2020 · Educating Text Autoencoders: Latent Representation Guidance via Denoising (ICML 2020 · 30 citations)
- 2022 · Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (CVPR 2022 · 315 citations)
- 2023 · Semantic-Conditional Diffusion Networks for Image Captioning (CVPR 2023 · 120 citations)
- 2024 · Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · 28 citations)
- 2025 · Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · 8 citations)
- 2026 · Scalable Single-Cell Gene Expression Generation with Latent Diffusion Models (ICML 2026 · 2 citations)
Earliest papers here
- Uncorrelation and Evenness: a New Diversity-Promoting Regularizer (ICML 2017 · 18 citations)
- Nonlocal Neural Networks, Nonlocal Diffusion and Nonlocal Modeling (NeurIPS 2018 · 29 citations)
- Semantic Interpolation in Implicit Models (ICLR 2018 · 3 citations)
What has been tried
64 papers here report 696 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 6%, hurt 23%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use continuous time rather than discrete T=1000 | LM1B test · PPL | 27.19 ± .07 -> 27.04 ± .01 | Helped | Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · 28 citations) |
| Use scheduled β instead of a constant β | Synthetic data quality: single-column density and pair-wise column correlation · error rate | Helped | Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space (ICLR 2024 · 23 citations) | |
| add the compact bottleneck | ImageNet-1K linear probe · Top1 | 34.2 -> 72.2 | Helped | SODA: Bottleneck Diffusion Models for Representation Learning (CVPR 2024 · 21 citations) |
| Increase visual prompts from 1 to 10 | COCO · mIoU | 49.7 -> 56.7 | Helped | Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · 8 citations) |
| Initialize NIVeL from random parameters rather than the RGB generator output | Text-to-vector generation · visual quality | Hurt | NIVeL: Neural Implicit Vector Layers for Text-to-Vector Generation (CVPR 2024 · 8 citations) | |
| Add classifier-free guidance (w=1.0) | NuScenes 3s · Chamfer (within ROI) | 1.40 -> 0.56; 60%↓ | Helped | Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion (ICLR 2024 · 7 citations) |
| add diffusion modelling as the decoder | Shapes3D · DCI | 0.178 ± 0.050 -> 0.969 ± 0.030 | Helped | Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement (NeurIPS 2024 · 6 citations) |
| Align to higher-quality pretrained visual representations | ImageNet 256×256, 400K iterations · FID | 12.5 -> 10.0 | Helped | Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · 4 citations) |
| Add content-mask factorization to the encoder-generator baseline | MPI3D-real-complex · DCI | 0.506 -> 0.631 | Helped | Factorized Diffusion Autoencoder for Unsupervised Disentangled Representation Learning (AAAI 2024 · 4 citations) |
| add Representation Guidance | ImageNet 256×256, conditional generation, 400K steps · FID | 8.7 -> 5.9 | Helped | Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · 3 citations) |
| Replace residual VQ with a single vanilla VQ layer | Conditional Kitchen, unconditional Ant, and nuScenes · performance | Hurt | Behavior Generation with Latent Actions (ICML 2024 · 3 citations) | |
| Increase the hierarchy from one level to five levels | ImageNet-1K 256×256, unconditional, w/o CFG · FID | 55.41 -> 11.05 | Helped | Nested Diffusion Models Using Hierarchical Latent Priors (CVPR 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 38 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet · 8 of 38 papers
- ImageNet-1K · 6 of 38 papers
- CelebA-HQ · 5 of 38 papers
- COCO · 3 of 38 papers
- OpenWebText · 3 of 38 papers
- ShapeNet · 3 of 38 papers
- FFHQ · 3 of 38 papers
- ROCStories · 2 of 38 papers
Baselines
- REPA · 7 of 38 papers
- DiT-XL/2 · 6 of 38 papers
- SiT-XL/2 · 5 of 38 papers
- VA-VAE · 4 of 38 papers
- SD-VAE · 3 of 38 papers
- RAE · 3 of 38 papers
- DiffuSeq · 2 of 38 papers
- SeqDiffuSeq · 2 of 38 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 105 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 5 papers
- Google · 5 papers
- Constructor University · 4 papers
- Amazon · 4 papers
- Tsinghua University · 4 papers
- Higher School of Economics · 4 papers
- Massachusetts Institute of Technology · 4 papers
- Fudan University · 3 papers
- Peking University · 3 papers
- Adobe · 3 papers
Use it
Download BibTeX of 34 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Generative latent space tooling (129 papers)
- High-resolution diffusion synthesis (140 papers)
- Hierarchical VQ-VAE generation (62 papers)
- Next-scale and token autoregressive images (90 papers)
- Accelerating autoregressive image generation (22 papers)
Elsewhere in the topic
- Classical generative image synthesis (147 papers)
- Retrieval and matching with generative features (140 papers)
- Generalizable fake image detection (119 papers)
- Synthetic training data from generators (111 papers)
- Compositional scene generation (105 papers)
- Robust diffusion watermarking (79 papers)
- Neural text-to-speech synthesis (72 papers)
- Equivariant diffusion for 3D molecules (71 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).