Research map / Image generation / Autoregressive and latent generation
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual AutoRegressive (VAR) modeling, which replaces raster-scan next-token prediction with coarse-to-fine next-scale prediction over multi-scale VQ token maps, letting GPT-style transformers surpass DiT on ImageNet generation and show power-law scaling.
Problem
Autoregressive image generation with raster-scan next-token prediction lags diffusion models in quality, is computationally expensive, and breaks the unidirectional dependency assumption on 2D image tokens.
Approach
Define the autoregressive unit as an entire token map: encode an image into multi-scale token maps and predict each higher-resolution map conditioned on all coarser ones, with a block-wise causal attention mask in training and a multi-scale residual quantizer with shared codebook.
Key results
- VAR-d30 with rejection sampling reaches FID 1.73 and IS 350.2 on ImageNet 256x256, versus 18.65 FID for the VQGAN AR baseline.
- VAR-d30 (2.0B) outperforms DiT-XL/2 (FID 2.27) and L-DiT-3B/7B (FID 2.10/2.28) while running about 45x faster than DiT-XL/2.
- Test loss and token error rate follow power laws in model size and compute, with Pearson correlation near -0.998.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace AR next-token prediction with VAR next-scale prediction (ablation) | AR transformer · ImageNet 256×256 | FID: 18.65 -> 5.22 | Helped |
| replace AR next-token prediction with VAR next-scale prediction (ablation) | AR transformer · ImageNet 256×256 | inference cost relative to baseline: 1 -> 0.013 | Helped |
| add AdaLN (ablation) | VAR-d16 · ImageNet 256×256 | FID: 5.22 -> 4.95 | Helped |
| add Top-k sampling (k=600) (ablation) | VAR-d16 + AdaLN · ImageNet 256×256 | FID: 4.95 -> 4.64 | Helped |
| add CFG with ratio 2.0 (ablation) | VAR-d16 + AdaLN + Top-k · ImageNet 256×256 | FID: 4.64 -> 3.60 | Helped |
| normalize q and k to unit vectors before attention (ablation) | VAR-d16 + AdaLN + Top-k + CFG · ImageNet 256×256 | FID: 3.60 -> 3.30 | Helped |
| scale model up to 2.0B parameters (ablation) | VAR · ImageNet 256×256 | FID: 3.30 -> 1.73 | Helped |
| use VAR instead of the AR baseline | Image generation models · ImageNet 256×256 | FID: 2.10 -> 1.73 | Helped |
| use VAR instead of DiT-XL/2 | Image generation models · ImageNet 256×256 | IS: 278.2 -> 350.2 | Helped |
| use VAR instead of DiT-XL/2 | Image generation models · ImageNet 512×512 | FID: 3.04 -> 2.63 | Helped |
| increase VAR transformer size | VAR · ImageNet validation set | test cross-entropy loss | Helped |
| increase VAR model size and training compute | VAR · ImageNet 256×256 generated samples | visual fidelity and soundness | Helped |
Surprising findings
- Replacing raster-scan AR with next-scale prediction alone cuts FID from 18.65 to 5.22 at 0.013x the inference cost, before other enhancements.
Limitations
- Text-to-image generation and video generation are not implemented; they are stated as future work.
- The VQVAE tokenizer architecture and training are kept unchanged from the VQGAN baseline, so tokenizer improvements are left unexplored.
Experimental setup
Datasets: ImageNet OpenImages
Benchmarks: ImageNet 256x256 class-conditional generation ImageNet 512x512 class-conditional generation
Metrics: FID IS precision recall wall-clock inference time test cross-entropy loss token error rate
Baselines: VQGAN DiT-XL/2 L-DiT-3B L-DiT-7B MaskGIT ADM BigGAN StyleGAN-XL RQTransformer ViT-VQGAN
Builds on: VQGAN GPT-2 DiT MaskGIT
Closest papers by meaning
- FasterVAR: Plug-and-Play Acceleration for Visual Autoregressive Models (ICML 2026 · similarity 0.94)
- Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities (ICML 2025 · similarity 0.93)
- FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction (NeurIPS 2025 · similarity 0.92)
- TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generation (NeurIPS 2025 · similarity 0.91)
- Visual Autoregressive Modeling for Image Super-Resolution (ICML 2025 · similarity 0.91)
- HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.