Research map / Image generation / Text-to-image diffusion
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Speculative Jacobi Decoding (SJD), a training-free probabilistic parallel decoding method that lets sampling-based autoregressive text-to-image models accept multiple draft tokens per forward pass, achieving about 2x step and latency speedups.
Problem
Autoregressive text-to-image generation needs hundreds to thousands of sequential next-token steps, and existing training-free Jacobi decoding uses a deterministic convergence criterion that only works with greedy decoding, not the top-K sampling needed for diverse, high-quality images.
Approach
Replace Jacobi decoding's deterministic convergence check with a speculative-sampling-style probabilistic acceptance rule that compares conditional probabilities from consecutive iterations, resamples rejected tokens from a calibrated residual distribution, and reuses unaccepted tokens as refinement drafts; additionally initialize new draft tokens from spatially adjacent tokens.
Key results
- On Lumina-mGPT 7B (MS-COCO, A100), SJD with spatial-prior initialization reaches 2.05x latency and 2.23x step acceleration with FID 31.13 vs 30.76 baseline, while vanilla Jacobi decoding gives only about 1.02x.
- On Anole 7B, SJD with spatial prior achieves 1.87x latency and 1.97x step acceleration on MS-COCO, and 1.92x latency and 2.11x step on Parti-prompts.
- Ablation with 1024x1024 generation shows SJD reaches 2.43x step acceleration, and LlamaGen acceleration is lower at about 1.5-1.7x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace deterministic Jacobi decoding with probabilistic SJD | Lumina-mGPT 7B · MSCOCO2017 val | latency speedup: $1.02 \times$ -> $2.04 \times$ | Helped |
| Replace deterministic Jacobi decoding with probabilistic SJD | Lumina-mGPT 7B · Parti-prompt val | latency speedup: $1.01 \times$ -> $2.12 \times$ | Helped |
| Apply SJD with spatial-prior token initialization | Anole 7B · MSCOCO2017 val | latency speedup: $1.00 \times$ -> $1.87 \times$ | Helped |
| Apply SJD | LlamaGen-stage1 · MSCOCO2017 | step speedup: $1.00 \times$ -> $1.63 \times$ | Helped |
| Apply SJD | LlamaGen-stage2 · Parti-prompt | step speedup: $1.00 \times$ -> $1.69 \times$ | Helped |
| Add spatial-prior token initialization to SJD (ablation) | Lumina-mGPT 7B · MSCOCO2017 val | latency speedup: $2.04 \times$ -> $2.05 \times$ | Helped |
| Add spatial-prior token initialization to SJD (ablation) | Lumina-mGPT 7B · Parti-prompt val | CLIP-Score: 32.13 -> 32.06 | Hurt |
| Increase image resolution from 768 × 768 to 1024 × 1024 (ablation) | SJD on Lumina-mGPT 7B · Lumina-mGPT image generation | step compression ratio: $2.43 \times$ at 1024 × 1024 | Helped |
| Increase Jacobi window size to at least 16 tokens (ablation) | SJD on Lumina-mGPT 7B · Lumina-mGPT image generation | acceleration ratio | Helped |
| Use spatial-prior initialization instead of random initialization (ablation) | SJD on Lumina-mGPT 7B · Images with simple repeating patterns | step compression ratio | Helped |
| Deactivate refinement and replace unaccepted tokens with newly initialized tokens (ablation) | SJD · Image generation | forward passes: over two thousand forward passes rather than one thousand forward passes | Hurt |
| Use deterministic Jacobi decoding instead of SJD under sampling randomness (ablation) | Lumina-mGPT · Top-K sampling | step compression ratio | Mixed |
Surprising findings
- Deactivating refinement of unaccepted tokens makes the model need over two thousand forward passes instead of about one thousand, so acceleration comes mainly from refining unaccepted tokens.
Limitations
- Acceleration on smaller LlamaGen models is lower, and failure cases show SJD can accept draft tokens that create artifacts.
Experimental setup
Datasets: MS-COCO 2017 validation set Parti-prompts
Benchmarks: MS-COCO 2017 validation Parti-prompts
Metrics: FID CLIP-Score HPSv2 step compression ratio latency speedup perplexity
Baselines: Lumina-mGPT autoregressive decoding Jacobi decoding Anole autoregressive decoding LlamaGen
Builds on: Jacobi decoding speculative sampling Lumina-mGPT Anole
Closest papers by meaning
- Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation (NeurIPS 2025 · similarity 0.94)
- Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation (NeurIPS 2025 · similarity 0.90)
- LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding (ICLR 2025 · similarity 0.90)
- Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient (CVPR 2025 · similarity 0.89)
- ZipAR: Parallel Autoregressive Image Generation through Spatial Locality (ICML 2025 · similarity 0.88)
- Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropy (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.