Research map / Image generation / Autoregressive and latent generation
ControlAR: Controllable Image Generation with Autoregressive Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ControlAR, a ViT-based control encoder plus per-token additive "conditional decoding" that adds spatial control tokens to image tokens in several layers of an autoregressive image generator, enabling controllable and arbitrary-resolution generation.
Problem
Adding spatial controls (edges, depth, segmentation) like ControlNet to autoregressive image generators is largely unexplored; naive prefilling of control tokens gives weak control quality and increases sequence length and cost.
Approach
Instead of prefilling control tokens into the sequence, fuse each control token by addition with the corresponding (shifted by one) image token in several intermediate sequence layers (conditional decoding), using a lightweight ViT control encoder, which keeps sequence length fixed and enables arbitrary-resolution generation by matching control token length.
Key results
- On ImageNet canny C2I at 256x256, LlamaGen-L with ControlAR reaches 7.69 FID, versus 10.85 FID for ControlNet and 26.45 for conditional prefilling with LlamaGen.
- On MultiGen-20M hed edges, ControlAR achieves 85.63 SSIM versus 80.97 for ControlNet++ (about +4.66) and 10.53 FID versus 15.01 for ControlNet++.
- On ADE20K segmentation, ControlAR gets 39.95 mIoU, below ControlNet++ at 43.64, but achieves the best COCOStuff mIoU of 37.49.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use conditional decoding instead of conditional prefilling (ablation) | LlamaGen-B ControlAR · ImageNet Canny (C2I) | F1-Score and FID | Helped |
| Replace cross-attention with addition for control fusion at the first layer (ablation) | LlamaGen-B ControlAR · ImageNet Canny (C2I) | F1-Score: 30.86 -> 34.01 | Helped |
| Replace cross-attention with addition for control fusion at the first layer (ablation) | LlamaGen-B ControlAR · ImageNet Canny (C2I) | FID: 15.34 -> 11.02 | Helped |
| Add control fusion at layers 1, 5, and 9 instead of only the first layer (ablation) | LlamaGen-B ControlAR · ImageNet Canny (C2I) | F1-Score: 34.01 -> 34.15 | Helped |
| Add control fusion at every layer instead of layers 1, 5, and 9 (ablation) | LlamaGen-B ControlAR · ImageNet Canny (C2I) | F1-Score and FID: 34.15 -> 34.21; 10.64 -> 11.75 | Mixed |
| Fully fine-tune the sequence model instead of freezing it (ablation) | LlamaGen-B ControlAR · ImageNet Canny (C2I) | F1-Score and FID: 30.62 -> 34.15; 13.67 -> 10.64 | Helped |
| Use LoRA instead of freezing the sequence model (ablation) | LlamaGen-B ControlAR · ImageNet Canny (C2I) | F1-Score and FID: 30.62 -> 32.90; 13.67 -> 13.20 | Helped |
| Use ViT-S instead of CNN control encoder (ablation) | ControlAR · ImageNet Canny (C2I) | F1-Score: 33.55 -> 34.15 | Helped |
| Use DINOv2-S instead of ViT-S control encoder (ablation) | ControlAR · MultiGen-20M Hed (T2I) | SSIM: 82.37 -> 85.63 | Helped |
| Use DINOv2-B instead of DINOv2-S control encoder (ablation) | ControlAR · MultiGen-20M Hed (T2I) | SSIM: 85.63 -> 86.12 | Helped |
| Add 2k fine-tuning iterations on ADE20K (ablation) | ControlAR segmentation model · ADE20K Segmentation (T2I) | mIoU: +1.15 | Helped |
Surprising findings
- Conditional prefilling of control tokens performs much worse than ControlNet (26.45 vs 10.85 FID) despite being the natural LLM-style approach.
- Lumina-mGPT (7B) performs poorly on controllable generation compared to ControlAR (0.8B) in the appendix comparison.
Limitations
- Full fine-tuning of the generative model is needed, which is less portable than ControlNet's plug-in design.
- Does not support multiple control images simultaneously.
- Text prompts can be overridden by strong spatial controls when they conflict.
Experimental setup
Datasets: ImageNet-1K MultiGen-20M ADE20K COCOStuff LAION-Aesthetics
Benchmarks: ImageNet-1K ADE20K COCOStuff MultiGen-20M
Metrics: FID F1-Score RMSE mIoU SSIM
Baselines: ControlNet ControlNet++ ControlVAR GLIGEN T2I-Adapter Uni-ControlNet UniControl OmniGen Lumina-mGPT
Builds on: LlamaGen AiM ControlNet DINOv2
Closest papers by meaning
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.92)
- HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation (CVPR 2025 · similarity 0.90)
- Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots (ICML 2025 · similarity 0.90)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.89)
- Simplifying Control Mechanism in Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.88)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.