Research map / Image generation / Autoregressive and latent generation
Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ST-AR, a self-guided training framework that adds masked image modeling on attention maps and inter-step/inter-view contrastive losses (with an EMA teacher) to autoregressive image generators, improving LlamaGen's ImageNet FID and linear probing without changing AR sampling.
Problem
Next-token prediction autoregressive image generators learn weak high-level visual representations: they rely on local/conditional context, are semantically inconsistent across generation steps, and lack invariance to visual tokenizer perturbations.
Approach
Adds self-supervised objectives during AR training: random masking of attention maps with an EMA-teacher feature alignment (MIM), plus inter-step and inter-view contrastive losses, forming an iBOT-style framework that needs no pretrained representation model and keeps the standard AR sampling strategy.
Key results
- ST-AR improves LlamaGen-B linear probing top-1 accuracy from 18.68% to 45.27% in the ablation (and 21.00% to 55.23% in the introduction).
- ST-AR gives about 42% FID improvement for LlamaGen-L (3.81 to 2.98) and about 49% for LlamaGen-XL at 50 epochs (3.39 to 2.72 in Table 2 with CFG).
- ST-AR LlamaGen-XL trained 50 epochs is said to be comparable to LlamaGen-3B trained 300 epochs despite about 4x fewer parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ST-AR training | LlamaGen-B · ImageNet-256×256 | FID: 5.46 -> 4.09 | Helped |
| Add ST-AR training | LlamaGen-L · ImageNet-256×256 | FID: 3.81 -> 2.98 | Helped |
| Add ST-AR training | LlamaGen-XL, 50 epochs · ImageNet-256×256 | FID: 3.39 -> 2.72 | Helped |
| Add MIM loss (ablation) | LlamaGen-B · ImageNet-256×256, 50 epochs | FID: 31.35 -> 30.58 | Helped |
| Add inter-step contrastive loss to MIM (ablation) | LlamaGen-B · ImageNet-256×256, 50 epochs | FID: 30.58 -> 28.02 | Helped |
| Add inter-view contrastive loss to MIM (ablation) | LlamaGen-B · ImageNet-256×256, 50 epochs | FID: 30.58 -> 27.78 | Helped |
| Add all three ST-AR losses (ablation) | LlamaGen-B · ImageNet-256×256, 50 epochs | FID: 31.35 -> 26.58 | Helped |
| Increase mask ratio from 0.25 to 0.35 (ablation) | LlamaGen-B + ST-AR · ImageNet-256×256 | FID: 26.58 -> 26.36 | Helped |
| Increase mask ratio from 0.25 to 0.45 (ablation) | LlamaGen-B + ST-AR · ImageNet-256×256 | sFID: 7.70 -> 8.31 | Hurt |
| Apply contrastive losses at depth 6 rather than depth 3 (ablation) | LlamaGen-B + ST-AR · ImageNet-256×256 | FID: 27.34 -> 26.58 | Helped |
| Apply contrastive losses at depth 9 rather than depth 6 (ablation) | LlamaGen-B + ST-AR · ImageNet-256×256 | FID: 26.58 -> 28.76 | Hurt |
| Increase selected steps from 2 to 4 (ablation) | LlamaGen-B + ST-AR · ImageNet-256×256 | FID: 27.50 -> 26.58 | Helped |
Surprising findings
- Masking attention maps rather than input tokens works for AR models, since masking input tokens would break next-token prediction.
Limitations
- Increased training cost from the auxiliary losses and EMA teacher.
- Gains depend on the LlamaGen/VQGAN tokenizer setup; evaluated only on ImageNet class-conditional generation.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-256x256
Metrics: FID sFID IS Precision Recall linear probing top-1 accuracy
Baselines: LlamaGen-B LlamaGen-L LlamaGen-XL VAR MaskGIT LDM-4 DiT-XL SiT-XL BigGAN StyleGAN-XL
Builds on: LlamaGen iBOT SimSiam MAE VQGAN
Closest papers by meaning
- Parallelized Autoregressive Visual Generation (CVPR 2025 · similarity 0.90)
- Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective (NeurIPS 2024 · similarity 0.90)
- Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots (ICML 2025 · similarity 0.90)
- OmniGen-AR: AutoRegressive Any-to-Image Generation (NeurIPS 2025 · similarity 0.89)
- Improving Autoregressive Visual Generation with Cluster-Oriented Token Prediction (CVPR 2025 · similarity 0.89)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.