Research map / Image generation / Autoregressive and latent generation
A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DnD-Transformer, an end-to-end autoregressive image generator that adds a depth (residual-code) autoregression direction via extra prediction heads, improving VQ image generation on ImageNet and enabling unconditional rich-text image generation.
Problem
VQ-based autoregressive image generation loses fine-grained information due to low information compression ratio, and increasing resolution or codebook size is computationally costly; prior 2D decompositions (RQ-Transformer) need separate transformers, hindering end-to-end LLM integration.
Approach
Introduce a depth autoregression direction over residual-quantized codes, predicted by multiple prediction heads attached to different layers of a single LLaMA-style decoder, so d codes per position are produced in one forward pass with no extra sequence length or modules.
Key results
- DnD-Transformer-XXL (cfg=2) reaches FID 2.58 and IS 295.6 on ImageNet 256x256 vs LlamaGen-XXL (cfg=2) FID 4.12 and IS 266.9 under the same implementation.
- DnD-Transformer-XXXL (2.5B, sequence length 256) with cfg=1.7 achieves FID 2.21, better than LlamaGen-3B (3.1B, sequence length 576) reported FID 4.21.
- Tokenizer with depth 8 reaches rFID 0.42 on ImageNet, versus 2.98 at depth 1 and 0.68 for SDXL VAE; DnD AR model outperforms DDPM and SD baselines on unconditional rich-text generation by OCR perplexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace 1D autoregression with 2D depth autoregression | DnD-Transformer-XXL vs. LlamaGen-XXL · ImageNet 256×256 | FID: 4.12 -> 2.58 | Helped |
| Increase residual code depth from 1 to 2 (ablation) | DnD visual tokenizer · ImageNet 256×256 validation | rFID: 2.98 -> 0.93 | Helped |
| Increase residual code depth from 2 to 4 (ablation) | DnD visual tokenizer · ImageNet 256×256 validation | rFID: 0.93 -> 0.60 | Helped |
| Increase residual code depth from 4 to 8 (ablation) | DnD visual tokenizer · ImageNet 256×256 validation | rFID: 0.60 -> 0.42 | Helped |
| Predict depth codes in parallel rather than using the full DnD autoregressive structure (ablation) | DnD-Transformer · ImageNet 256×256 | FID: 2.58 -> 6.32 | Hurt |
| Use sequential vertical prediction without autoregressive conditioning (ablation) | DnD-Transformer · ImageNet 256×256 | FID: 2.58 -> 3.18 | Hurt |
| Increase DnD code depth (ablation) | DnD-Transformer · Image generation | training cross-entropy loss | Helped |
| Increase DnD code depth (ablation) | DnD-Transformer · ImageNet generation | generation fidelity | No clear effect |
Surprising findings
- Deeper code prediction lowers training cross-entropy but does not improve ImageNet generation fidelity, attributed to a larger sampling space.
- Discrete VQ tokenizers with enough depth reconstruct rich text better than continuous SD3 and SDXL VAEs, contrary to the common criticism of VQ information loss.
Limitations
- Increasing depth does not improve ImageNet generation fidelity, with a larger multi-depth sampling space left unresolved.
- Rich-text evaluation relies on Qwen VLM OCR and LLM perplexity as judges, and models are limited in size and data compared to large language models.
Experimental setup
Datasets: ImageNet-1k Text-Image (rendered English Wikipedia, 2.4M) arXiv-Image (2M crops from arXiv PDFs)
Benchmarks: ImageNet 256x256 Text-Image arXiv-Image
Metrics: FID IS Precision Recall rFID rOCR (Rouge-L) PPL_ocr
Baselines: LlamaGen-XXL LlamaGen-3B RQTransformer ADM LDM-4 DiT-XL/2 VQGAN DDPM SDXL SD3
Builds on: RQ-Transformer RQ-VAE LlamaGen LLaMA VQGAN
Closest papers by meaning
- Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities (ICML 2025 · similarity 0.89)
- JetFormer: An autoregressive generative model of raw images and text (ICLR 2025 · similarity 0.88)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.88)
- HART: Efficient Visual Generation with Hybrid Autoregressive Transformer (ICLR 2025 · similarity 0.88)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.87)
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.