Research map / Large language models / Language, safety and interpretability
Elucidating the design space of language models for image generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically explores the design space of autoregressive language models for ImageNet 256x256 generation, using a binary autoencoder tokenizer with code decomposition, and yields ELM models reaching FID 1.54 with a 1.9B model.
Problem
Language models for image generation remain underexplored; image tokens are far more random than text tokens, and it is unclear which tokenizer, modeling method, vocabulary design, and sampling strategy work best when adapting LMs to vision.
Approach
Combines a binary autoencoder (look-up-free quantization with Bernoulli sampling) with decomposition of large binary codes into two subcodes and a plain AR LLaMA-style transformer with linear CFG scheduling and high top-k randomness, and analyzes attention to explain scaling.
Key results
- ELM-2B (2-12 tokenizer, 1.9B params) reaches FID 1.54 and IS 332.69 on ImageNet 256x256 class-conditional generation, the best reported in the comparison table.
- Binary autoencoder BAE-f16 achieves 100% code utilization versus about 8% for VQGAN-f16 and lower rFID (3.32 vs 7.41 at comparable settings).
- Splitting the 16-bit code into two 8-bit subcodes (2-8) or using 2-10/2-12 decompositions improves FID over the undecomposed 1-16 tokenizer for L, XL and XXL models (e.g., XL 1.73 with 2-10 vs 2.14 with 1-16).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace VQGAN-f16 with BAE-f16 as the tokenizer (ablation) | AR-L image generation · ImageNet 256×256 | gFID: 6.71 -> 2.78 | Helped |
| Add Bernoulli sampling during BAE quantization (ablation) | AR-L with BAE-f16 · ImageNet 256×256, constant CFG 2 | FID: 5.05 -> 4.72 | Helped |
| Add Bernoulli sampling during BAE quantization (ablation) | AR-L with BAE-f16 · ImageNet 256×256, linear CFG 1-3 | FID: 3.13 -> 2.88 | Helped |
| Use autoregressive modeling instead of MLM (ablation) | L, XL, and XXL models · ImageNet 256×256 | FID: L: 3.67 -> 2.38; XL: 3.13 -> 2.14; XXL: 3.12 -> 2.10 | Helped |
| Split the code into two subcodes rather than using one code (ablation) | AR-L with BAE · ImageNet 256×256 | FID: 2.38 -> 2.17 | Helped |
| Use two 10-bit subcodes instead of three 8-bit subcodes (ablation) | AR-XL with BAE · ImageNet 256×256 | FID: 1.99 -> 1.73 | Helped |
| Use two 12-bit subcodes instead of three or four 8-bit subcodes (ablation) | AR-XXL with BAE · ImageNet 256×256 | FID: 3-8: 1.67 -> 1.58; 4-8: 2.02 -> 1.58 | Helped |
| Increase decomposition from two to four subcodes (ablation) | AR-XXL with BAE · ImageNet 256×256 | FID: 1.58 -> 2.02 | Hurt |
| Use linear CFG scheduling instead of constant CFG 1.5 (ablation) | AR-L with BAE 2-10 · ImageNet 256×256 | FID: 2.98 -> 2.48 | Helped |
| Increase AR model size from L to XL to XXL to 2B | ELM · ImageNet 256×256 class-conditional | FID: 2.17 -> 1.79 -> 1.58 -> 1.54 | Helped |
Surprising findings
- Image training loss stays high and does not converge well, yet models still produce low-FID images, which the authors attribute to the near-uniform randomness of image tokens.
- Higher randomness in sampling (larger top-k, Bernoulli sampling) helps image generation, the opposite of the usual low-temperature preference for text.
Limitations
- Next-token classification loss may not be the optimal objective for highly random image tokens, as acknowledged by the authors.
- BAE tokenizer trained only on ImageNet, limiting generality to other image domains.
- The 2B model training was stopped early due to time constraints, so it is not fully converged.
Experimental setup
Datasets: ImageNet OpenWebText Shakespeare
Benchmarks: ImageNet 256x256
Metrics: FID IS Precision Recall sFID rFID
Baselines: VQGAN ViT-VQGAN RQTransformer VAR LlamaGen MAR
Builds on: VQGAN LlamaGen MAR LLaMA-2 Bernoulli sampling binary autoencoder
Closest papers by meaning
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.92)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.91)
- A Vision Check-up for Language Models (CVPR 2024 · similarity 0.91)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.90)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.90)
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.