Research map / Image generation / Autoregressive and latent generation
Hierarchical VQ-VAE generation
Generate images with hierarchical quantized or multi-scale latent representations.
62 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Autoregressive and latent generation, keyword cluster “latent space · modeling · vae”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
6 papers in 2023–24 and 27 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 1.3% to 3.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Towards Conceptual Compression (NeurIPS 2016 · 103 citations)
- 2017 · Learning Diverse Image Colorization (CVPR 2017 · 202 citations)
- 2018 · A Variational U-Net for Conditional Appearance and Shape Generation (CVPR 2018 · 508 citations)
- 2019 · Generating Diverse High-Fidelity Images with VQ-VAE-2 (NeurIPS 2019 · 105 citations)
- 2020 · ControlVAE: Controllable Variational Autoencoder (ICML 2020 · 49 citations)
- 2021 · Dual Contradistinctive Generative Autoencoder (CVPR 2021 · 67 citations)
- 2022 · Vector-quantized Image Modeling with Improved VQGAN (ICLR 2022 · 91 citations)
- 2023 · Catch Missing Details: Image Reconstruction With Frequency Augmented Variational Autoencoder (CVPR 2023 · 30 citations)
- 2024 · Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications (CVPR 2024 · 257 citations)
- 2025 · SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer (CVPR 2025 · 11 citations)
- 2026 · VAEVQ: Enhancing Discrete Visual Tokenization Through Variational Modeling (AAAI 2026 · 1 citation)
Earliest papers here
- Towards Conceptual Compression (NeurIPS 2016 · 103 citations)
- An Architecture for Deep, Hierarchical Generative Models (NeurIPS 2016 · 25 citations)
- Learning Diverse Image Colorization (CVPR 2017 · 202 citations)
What has been tried
42 papers here report 450 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 7%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Remove softmax normalization from spatial aggregation weights | ImageNet-1K classification · convergence speed and final performance | Helped | Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications (CVPR 2024 · 257 citations) | |
| use restrictive encoder and partial-convolution decoder instead of the vanilla network designs | Places Box (3000-image subset) · FID | 15.32; 18.50 -> 12.81 | Helped | Don't Look into the Dark: Latent Codes for Pluralistic Image Inpainting (CVPR 2024 · 17 citations) |
| Add DINOv2-B feature alignment and encoder initialization | ImageNet-1K 256×256 · gFID | 20.33 -> 10.13 | Helped | SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer (CVPR 2025 · 11 citations) |
| Add two attention blocks to the jointly trained decoder | VQ tokenizer with AR transformer · generation FID | 16.97 -> 17.42 | Hurt | Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis (CVPR 2024 · 8 citations) |
| Increase codebook count and expansion value | 20NG · TQ | No clear effect | Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation (AAAI 2024 · 7 citations) | |
| Increase MergeVQ (G) kept tokens from 64 to 256 | ImageNet-1K reconstruction · rFID | 4.74 -> 1.41 | Helped | MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization (CVPR 2025 · 4 citations) |
| add vectorized sparsity controller inference | BSDS500 test · SSIM | 0.87 -> 0.90 | Helped | SSC-VAE: Structured Sparse Coding Based Variational Autoencoder for Detail Preserved Image Reconstruction (AAAI 2025 · 3 citations) |
| Apply recurrent encoder processing for 2–3 iterations instead of one | ImageNet-1K · Top-1 accuracy | 55.6\% -> 59.5\% | Helped | Adaptive Length Image Tokenization via Recurrent Allocation (ICLR 2025 · 1 citation) |
| increase bottleneck E (latent floats) at fixed patch/FLOPs | ImageNet-1K 256p reconstruction · rFID, rPSNR, rSSIM | strong log(E) correlation with reconstruction quality | Helped | Learnings from Scaling Visual Tokenizers for Reconstruction and Generation (ICML 2025 · 0 citations) |
| Use rotation transformations only | ImageNet 256×256 · gFID | 43.5 -> 41.2 | Helped | EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling (ICML 2025 · 0 citations) |
| Replace standard diffusion parameterization with rectified flow | ImageNet 128×128 · rFID | 28.22 -> 24.11 | Helped | Epsilon-VAE: Denoising as Visual Decoding (ICML 2025 · 0 citations) |
| Remove 3D regularization | Objaverse Cake and Figurine NVS · PSNR | 34.38 -> 32.09; 33.17 -> 30.94 | Hurt | Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder (ICLR 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 15 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1K · 9 of 15 papers
- ImageNet · 4 of 15 papers
- COCO-2017 · 2 of 15 papers
- Megalith-10M · 2 of 15 papers
- BLIP3o-60k · 2 of 15 papers
Baselines
- MaskGIT · 6 of 15 papers
- VQGAN · 5 of 15 papers
- LlamaGen · 5 of 15 papers
- SD-VAE · 5 of 15 papers
- SDXL-VAE · 3 of 15 papers
- SiT · 3 of 15 papers
- RAE · 3 of 15 papers
- VQ-VAE · 2 of 15 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 33 papers; from OpenAlex and the papers' affiliations).
- Fudan University · 3 papers
- Zhejiang University · 3 papers
- National University of Singapore · 2 papers
- Google · 2 papers
- Google DeepMind · 2 papers
- UCAS/EPFL · 2 papers
- JD Explore Academy · 2 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Generative latent space tooling (129 papers)
- High-resolution diffusion synthesis (140 papers)
- Latent diffusion with learned tokenizers (114 papers)
- Next-scale and token autoregressive images (90 papers)
- Accelerating autoregressive image generation (22 papers)
Elsewhere in the topic
- Classical generative image synthesis (147 papers)
- Retrieval and matching with generative features (140 papers)
- Generalizable fake image detection (119 papers)
- Synthetic training data from generators (111 papers)
- Compositional scene generation (105 papers)
- Robust diffusion watermarking (79 papers)
- Neural text-to-speech synthesis (72 papers)
- Equivariant diffusion for 3D molecules (71 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).