Research map / Large language models / Language, safety and interpretability
Improving Autoregressive Visual Generation with Cluster-Oriented Token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IAR, which rearranges the VQGAN codebook into balanced k-means clusters and adds a cluster-oriented cross-entropy loss so LLaMA-style autoregressive image generators predict the correct cluster, improving training efficiency and FID on ImageNet 256.
Problem
Autoregressive LLM-based visual generation treats image tokens like words, ignoring that nearby codebook embeddings represent similar image content; a wrong token prediction can therefore produce artifacts and slow training.
Approach
Rearrange the VQGAN codebook via balanced k-means so that contiguous index ranges form clusters of visually similar codes, then add a cluster-level cross-entropy term (via LogSumExp over each cluster) so the model is rewarded for predicting the correct cluster even when the exact token is wrong.
Key results
- IAR-XXL (1.4B, CFG=1.7) achieves FID 2.19 and IS 278.9 on ImageNet 256, versus LlamaGen-XXL's 2.34 FID.
- IAR-B's 175-epoch model reaches roughly the FID of the 300-epoch LlamaGen-B, described as about 42% training acceleration.
- Ablation on cluster size shows best FID (6.77) at cluster size 128 (sqrt of 16384 codebook) with lambda=1 for the cluster loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add codebook rearrangement and LCCE (ablation) | LlamaGen-B / IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 6.77 | Helped |
| Add codebook rearrangement (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 7.15 | No clear effect |
| Add LCCE (ablation) | IAR-B · ImageNet, 100 epochs | FID: 6.962 -> 6.77 | Helped |
| Set cluster size to 128 (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 6.77 | Helped |
| Set cluster size to 8 (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 7.02 | Helped |
| Set cluster size to 16 (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 6.99 | Helped |
| Set cluster size to 32 (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 6.83 | Helped |
| Set cluster size to 64 (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 6.92 | Helped |
| Set cluster size to 256 (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 6.81 | Helped |
| Set cluster size to 512 (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 6.94 | Helped |
| Set LCCE weight λ to 0.1 (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 7.01 | Helped |
| Set LCCE weight λ to 0.25 (ablation) | IAR-B · ImageNet, 100 epochs | FID: 7.14 -> 7.05 | Helped |
Surprising findings
- Replacing codes with nearby codes (within code distance ~12) leaves decoded VQGAN images nearly identical, suggesting token-index errors are often visually benign.
Limitations
- Evaluation limited to ImageNet class-conditional generation at 256 and 384 resolution.
- Cluster size and loss weight are hyperparameters tuned by ablation on IAR-B.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 class-conditional generation
Metrics: FID IS Precision Recall
Baselines: LlamaGen VQGAN VQGAN-re ViT-VQGAN RQTransformer MaskGIT DiT-XL/2 LDM-4 BigGAN GigaGAN StyleGAN-XL ADM
Builds on: LlamaGen VQGAN classifier-free guidance balanced k-means
Closest papers by meaning
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.89)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.89)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.89)
- TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generation (NeurIPS 2025 · similarity 0.89)
- Language Model Beats Diffusion - Tokenizer is key to visual generation (ICLR 2024 · similarity 0.88)
- Parallelized Autoregressive Visual Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.