Research map / Image generation / Autoregressive and latent generation
Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes pre-trained autoregressive text-to-image models to denoise noise-perturbed token embeddings (next-clean-token prediction), then decodes with a Jacobi-denoising window plus probabilistic acceptance to cut forward passes about 4-5.6x.
Problem
Autoregressive text-to-image models decode token by token, requiring thousands of forward passes per image; existing parallel methods such as speculative Jacobi decoding (SJD) have unconstrained token refinement, leading to slow convergence.
Approach
Integrate the diffusion denoising trajectory into Jacobi decoding by fine-tuning the AR model with Gaussian-perturbed normalized token embeddings and a next-clean-token cross-entropy objective, so unaccepted tokens are refined by denoising steps (with timestep tokens) before standard Jacobi updates.
Key results
- On COCO2017, SJD2 reduces average steps for Lumina-mGPT from 2357 to 592 (4.02x step compression) with FID 31.4 vs 30.8 for the baseline.
- On Emu3, SJD2 achieves 5.62x step compression (1461 steps vs 8193) with FID 31.5 vs 31.1 for the baseline.
- On a subset of COCO prompts on one RTX 4090, SJD2 cuts Lumina-mGPT latency from 88.55s to 33.64s and Emu3 latency from 375.29s to 147.65s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SJD2 instead of autoregressive decoding | Lumina-mGPT · COCO2017 (5k) | Step Compression: $1.00\times$ -> $\mathbf{4.02\times}$ | Helped |
| Use SJD2 instead of SJD | Lumina-mGPT · COCO2017 (5k) | Step Compression: $2.23\times$ -> $\mathbf{4.02\times}$ | Helped |
| Use SJD2 instead of autoregressive decoding | Emu3 · COCO2017 validation subset | Latency: 375.29s -> 147.65s | Helped |
| Use SJD2 instead of autoregressive decoding | Lumina-mGPT · COCO2017 validation subset | Latency: 88.55s -> 33.64s | Helped |
| Use SJD2 instead of tuned autoregressive decoding | Lumina-mGPT · GenEval | Overall: 0.52 -> 0.51 | Hurt |
| Use SJD2 instead of tuned autoregressive decoding | Emu3 · GenEval | Overall: 0.52 -> 0.49 | Hurt |
| Use SJD2 instead of tuned autoregressive decoding | Janus-Pro-1B · GenEval | Overall: 0.59 -> 0.60 | Helped |
| Add embedding normalization to the denoising process (ablation) | SJD2 · Image generation | Visual coherence | Helped |
| Constrain denoising steps to 20 and use Jacobi window lengths beyond 80 (ablation) | SJD2 · Latency | Latency | Helped |
Surprising findings
- Latency gains are not consistent across models despite similar step compression, which the authors attribute to differing KV cache sizes.
- Pure denoising without Jacobi iterations can generate reasonable images, though with some quality loss.
Limitations
- Latency improvements are inconsistent across models, likely due to KV cache size differences.
- Additional GPU memory overhead of about 3GB and higher FLOPs per token than autoregressive decoding.
- Fine-tuning requires 8 GPUs with 80GB memory per model and tens of GPU-hours.
Experimental setup
Datasets: MSCOCO 2014 MSCOCO 2017 GenEval prompts self-collected 80k high-resolution images
Benchmarks: MS-COCO 2017 val (5k) MS-COCO 2014 val (30k) GenEval
Metrics: FID CLIP-Score step compression ratio latency GenEval score GPU memory
Baselines: Lumina-mGPT SJD Emu3 Jacobi Decoding EAGLE LANTERN ZipAR Distilled-Decoding
Builds on: Jacobi decoding Speculative Jacobi Decoding speculative sampling flow matching
Closest papers by meaning
- Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding (ICLR 2025 · similarity 0.94)
- Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation (NeurIPS 2025 · similarity 0.90)
- Epsilon-VAE: Denoising as Visual Decoding (ICML 2025 · similarity 0.90)
- Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropy (NeurIPS 2025 · similarity 0.89)
- LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding (ICLR 2025 · similarity 0.89)
- Parallelized Autoregressive Visual Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.