Research map / Image generation / Autoregressive and latent generation
Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiGIT, a discriminative image tokenizer that applies K-Means to DINOv2 features to build a stable discrete latent space for GPT-style image autoregressive modeling, improving both linear-probe understanding and ImageNet generation.
Problem
Autoregressive image models on reconstructive VQGAN latent spaces lag behind latent diffusion and mask image models in generation, suggesting the reconstruction-optimized latent space is unstable and hurts next-token generation.
Approach
Replace the reconstructive autoencoder latent space with discrete tokens obtained by K-Means clustering on frozen self-supervised (DINOv2) features, which are more stable to pixel perturbations; then train a GPT-2-style autoregressive transformer on these tokens and use a separate VQGAN or MaskGIT decoder to render pixels.
Key results
- DiGIT-732M reaches 80.3% top-1 linear-probe accuracy on ImageNet, versus 71.7% for the 219M model.
- Class-conditional ImageNet 256x256 generation: DiGIT (+VQGAN) 732M reaches FID 3.39 and IS 205.96 with classifier-free guidance-free sampling reported as state of the art for autoregressive models.
- Class-unconditional generation: DiGIT (+VQGAN) 732M achieves FID 4.59 versus 24.38 for the VQGAN AR baseline at 214M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the VQGAN tokenizer with the discriminative tokenizer (ablation) | autoregressive image generation · ImageNet 256×256 class-unconditional generation | FID: reducing FID to $9.66 | Helped |
| replace the VQGAN tokenizer with the discriminative tokenizer (ablation) | autoregressive image generation · ImageNet 256×256 class-unconditional generation | Inception Score: increasing IS to $69.15 | Helped |
| extend training duration to 400 epochs (ablation) | DiGIT autoregressive image generation · ImageNet 256×256 class-unconditional generation | FID: +0.53 | Helped |
| increase the model size to 732M (ablation) | DiGIT autoregressive image generation · ImageNet 256×256 class-unconditional generation | FID: FID dropping dramatically to $4.59$ | Helped |
| increase the K-Means cluster number (ablation) | DiGIT image understanding · ImageNet linear probe | Top-1 accuracy | Helped |
| use features from different transformer layers (ablation) | DiGIT image understanding · ImageNet linear probe | Top-1 accuracy | Mixed |
| use the discriminative tokenizer instead of the VQGAN tokenizer | autoregressive image generation · ImageNet 256×256 generation with a truncated prefix | FID | Helped |
Surprising findings
- Lower reconstruction FID of VQGAN does not imply better generation FID, challenging the assumption that good reconstruction gives a good latent space for generation.
- Under noise perturbation, discriminative (DINOv2-based) tokens change less than VQ tokens and keep higher cosine similarity, indicating greater latent stability.
Limitations
- The discriminative tokenizer cannot directly render pixels, so a separate VQGAN or MaskGIT decoder is required.
- Results for generation depend on training a second-stage decoder that maps discriminative tokens to VQ tokens.
- Evaluation is limited to ImageNet at 256x256 resolution.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K linear probe ImageNet 256x256 class-unconditional generation ImageNet 256x256 class-conditional generation
Metrics: Top-1 accuracy FID Inception Score
Baselines: VQGAN MaskGIT MAGE LDM GIVT VAR VIM AIM iGPT
Builds on: VQGAN DINOv2 MaskGIT GPT-2 iGPT
Closest papers by meaning
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.90)
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.90)
- Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots (ICML 2025 · similarity 0.89)
- ImageFolder: Autoregressive Image Generation with Folded Tokens (ICLR 2025 · similarity 0.89)
- Autoregressive Image Generation without Vector Quantization (NeurIPS 2024 · similarity 0.89)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.