Research map / Image generation / Autoregressive and latent generation
When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies the compression-generation trade-off in two-stage discrete autoregressive image generation via scaling laws, and proposes Causally Regularized Tokenization (CRT), which adds a small causal transformer next-token L2 loss on pre-quantized VQGAN latents to improve stage-2 generation at fixed rate.
Problem
Two-stage image generation trades off stage-1 reconstruction/compression against how easily stage-2 autoregressive models can learn the latent distribution; the optimal tokenizer compression depends on generative model capacity and compute.
Approach
Embed an autoregressive inductive bias into the stage-1 tokenizer by backpropagating an L2 next-token prediction loss from a small causal transformer (same architecture as stage 2) through the encoder before quantization, improving stage-2 scaling without changing bits per pixel.
Key results
- CRT_opt-AR-775M reaches 2.18 gFID on ImageNet 256x256 with 256 tokens per image, matching LlamaGen-3B (2.18 FID at 576 tokens, 3.1B params) with 775M parameters.
- CRT improves stage-2 compute efficiency by roughly 1.5-3x over the VQGAN baseline, with scaling exponent improving from -0.65 to -0.73, despite worse rFID (2.36 vs 2.21).
- On LSUN Cats, Horses, and Bedrooms, CRT improves gFID/gFID_CLIP over a tuned VQGAN baseline across 111M-775M models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add causal ℓ2 next-token regularization before quantization to the tokenizer | VQGAN tokenizer and autoregressive image generation · ImageNet, compute-controlled at 775M | gFID: 2.55 -> 2.35 | Helped |
| Add causal ℓ2 next-token regularization before quantization to the tokenizer | VQGAN tokenizer and autoregressive image generation · ImageNet 256×256 | rFID: 2.21 vs 2.36 | Hurt |
| Add causal ℓ2 next-token regularization before quantization to the tokenizer | VQGAN tokenizer and autoregressive image generation · LSUN Cats | gFID_CLIP: 7.37 -> 6.77 | Helped |
| Add causal ℓ2 next-token regularization before quantization to the tokenizer | VQGAN tokenizer and autoregressive image generation · LSUN Horses | gFID_CLIP: 4.92 -> 4.05 | Helped |
| Add causal ℓ2 next-token regularization before quantization to the tokenizer | VQGAN tokenizer and autoregressive image generation · LSUN Bedrooms | gFID_CLIP: 9.61 -> 8.56 | Helped |
| Use CRT_opt instead of the baseline tokenizer | 775M autoregressive text-to-image model · BLIP-3o pre-training dataset | gFID: 4.61 -> 4.15 | Helped |
| Use CRT_opt instead of the baseline tokenizer | 775M autoregressive text-to-image model · BLIP-3o pre-training dataset | CLIPScore: 0.33 -> 0.36 | Helped |
| Use ℓ2 next-token loss before quantization rather than after quantization (ablation) | CRT tokenizer · ImageNet 256×256 | Avg. gFID: 6.37 -> 5.51 | Helped |
| Use ℓ2 next-token loss before quantization rather than cross-entropy after quantization (ablation) | CRT tokenizer · ImageNet 256×256 | Avg. gFID: 6.10 -> 5.51 | Helped |
| Use ℓ2 next-token loss before quantization rather than LARP (ablation) | CRT tokenizer · ImageNet 256×256 | Avg. gFID: 6.08 -> 5.51 | Helped |
| Increase causal regularizer depth beyond 4 layers (ablation) | CRT tokenizer · ImageNet | average gFID | Hurt |
| Set the causal loss weight to λ = 4 rather than the baseline λ = 0 (ablation) | CRT tokenizer · ImageNet | generation FID | Helped |
Surprising findings
- Smaller stage-2 models benefit from more compressed tokenizers even when reconstruction worsens, and the 16k codebook beats both 1k and 131k despite 131k having better rFID.
- Cross-entropy causal loss after quantization hurts reconstruction and gives little generation gain, while L2 before quantization works better.
Limitations
- Largest stage-2 model is 775M to avoid overfitting on the finite ImageNet dataset, so results are in the repeated-data regime.
- Evaluation focuses on class-conditional ImageNet; text-to-image is only a single small-scale experiment, and LSUN FID is noisy due to out-of-distribution Inception features.
- Reordering tokens (non-raster order) is not improved by CRT; VQGAN architecture limits flexibility.
Experimental setup
Datasets: ImageNet LSUN BLIP-3o long-caption pretraining dataset
Benchmarks: ImageNet 256x256 LSUN Cats LSUN Horses LSUN Bedrooms BLIP-3o pretraining set
Metrics: gFID rFID IS precision recall PSNR MS-SSIM CLIPScore gFID_CLIP
Baselines: LlamaGen-3B LlamaGen-XL VAR MAR-H DiT-XL/2 LARP FSQ MaskGIT
Builds on: VQGAN LlamaGen VQ-VAE Llama 2 LARP
Closest papers by meaning
- Learnings from Scaling Visual Tokenizers for Reconstruction and Generation (ICML 2025 · similarity 0.92)
- Epsilon-VAE: Denoising as Visual Decoding (ICML 2025 · similarity 0.90)
- An Image is Worth 32 Tokens for Reconstruction and Generation (NeurIPS 2024 · similarity 0.90)
- ImageFolder: Autoregressive Image Generation with Folded Tokens (ICLR 2025 · similarity 0.90)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.89)
- Improving Autoregressive Visual Generation with Cluster-Oriented Token Prediction (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.