Research map / Image generation / Autoregressive and latent generation
Autoregressive Image Generation without Vector Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diffusion Loss, a small per-token diffusion head that lets autoregressive and masked autoregressive (MAR) image generators model continuous-valued tokens without vector quantization.
Problem
Autoregressive image generation is typically tied to discrete vector-quantized tokens, which are hard to train and reconstruct less well than continuous tokenizers; the paper asks whether AR modeling truly requires discrete tokens.
Approach
Replace the categorical cross-entropy loss over tokens with a small diffusion MLP conditioned on the AR output vector z, trained jointly with the autoregressive transformer to model each token's distribution p(x|z); combined with bidirectional-attention random-order "masked autoregressive" (MAR) generation that predicts multiple tokens per step.
Key results
- MAR-H with Diffusion Loss reaches FID 1.55 with CFG (2.35 without CFG) on ImageNet 256x256, competitive with leading systems.
- Diffusion Loss beats cross-entropy with the same tokenizer and architecture; MAR-L FID drops from 8.79 (CE, VQ-16) to 3.50 (Diff Loss, KL-16) without CFG, a roughly 50-60% relative reduction.
- MAR generates at under 0.3 s per image with FID under 2.0 on ImageNet 256x256 per the paper's speed/accuracy comparison against DiT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace cross-entropy on discrete VQ-16 tokens with Diff Loss on continuous KL-16 tokens | AR raster causal · ImageNet 256×256 | FID: 19.58 -> 4.69 | Helped |
| Replace cross-entropy on discrete VQ-16 tokens with Diff Loss on continuous KL-16 tokens | MAR random causal · ImageNet 256×256 | FID: 16.22 -> 13.07 | Helped |
| Replace cross-entropy on discrete VQ-16 tokens with Diff Loss on continuous KL-16 tokens | MAR random bidirectional · ImageNet 256×256 | FID: 8.75 -> 3.43 | Helped |
| Replace cross-entropy on discrete VQ-16 tokens with Diff Loss on continuous KL-16 tokens | MAR default · ImageNet 256×256 | FID: 8.79 -> 3.50 | Helped |
| Replace raster order with random order (ablation) | AR with Diff Loss · ImageNet 256×256, without CFG | FID: 19.23 -> 13.07 | Helped |
| Replace causal attention with bidirectional attention (ablation) | Random-order AR with Diff Loss · ImageNet 256×256, without CFG | FID: 13.07 -> 3.43 | Helped |
| Predict multiple tokens per step instead of one (ablation) | Random-order bidirectional MAR with Diff Loss · ImageNet 256×256 | FID: 3.43 -> 3.50 | Hurt |
| Use a stride-8 KL tokenizer and group 2×2 tokens (ablation) | MAR-L with Diff Loss · ImageNet 256×256 | FID: 1.98 -> 2.05 | Hurt |
| Replace KL-16 with a VQ-16 tokenizer, using its continuous latent before quantization (ablation) | MAR-L with Diff Loss · ImageNet 256×256, without CFG | FID: 3.50 -> 7.82 | Hurt |
| Replace random token order with on-the-fly confidence-based ordering (ablation) | MAR with cross-entropy · ImageNet 256×256 | FID: 8.79 -> 8.72 | No clear effect |
| Increase denoising MLP width (ablation) | MAR-L with Diff Loss · ImageNet 256×256 | FID | Helped |
| Use L2 loss instead of Diff Loss (ablation) | MAR · ImageNet | FID: >100 | Hurt |
Surprising findings
- An L2 (MSE) loss on continuous tokens yields FID above 100, so a stochastic per-token distribution model is essential.
- Diffusion Loss applied directly to VQ-16 pre-quantization latents (7.82 FID) beats cross-entropy on the same VQ tokenizer (8.79 FID).
Limitations
- Relies on pre-trained tokenizers whose quality caps generation quality, and visible artifacts remain.
- Evaluated mainly on ImageNet at academic scale; scalability to large text-to-image or video settings is untested.
- Pixel-space variant on ImageNet 64x64 is far more computationally expensive for higher resolutions.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 ImageNet 512x512 ImageNet 64x64
Metrics: FID IS precision recall rFID inference time
Baselines: ADM VDM++ MaskGIT MAGE MAGVIT-v2 LDM-4 DiT-XL/2 U-ViT-H/2-G DiffiT MDTv2-XL/2 GIVT
Builds on: DDPM MaskGIT MAE LDM classifier-free guidance
Closest papers by meaning
- Continuous Visual Autoregressive Generation via Score Maximization (ICML 2025 · similarity 0.89)
- Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective (NeurIPS 2024 · similarity 0.89)
- Autoregressive Video Generation without Vector Quantization (ICLR 2025 · similarity 0.88)
- Parallelized Autoregressive Visual Generation (CVPR 2025 · similarity 0.88)
- Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots (ICML 2025 · similarity 0.88)
- Denoising Autoregressive Transformers for Scalable Text-to-Image Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.