Research map / Image generation / Text-to-image diffusion
Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Hawk applies speculative decoding to autoregressive text-to-image generation using dual-direction (horizontal and vertical) draft heads that merge into a spatial sampling pool, achieving 1.71× speedup on Lumina-mGPT.
Problem
Autoregressive image generation is slow due to sequential token-by-token decoding, and speculative decoding designed for text does not exploit the 2D spatial structure of images or the much larger image token sampling space.
Approach
Add vertical draft heads that predict tokens one or more rows below alongside horizontal draft heads, cache vertical predictions for later rows, and merge both into a 2D spatial candidate pool verified by tree-based speculative sampling, so the residual rejection probability decreases with multiple distinct draft distributions.
Key results
- Hawk (spatial draft heads) achieves 1.71× speedup with accepted length 1.890 on COCO2017, versus 1.58× for Medusa and 1.69× for LANTERN++.
- On COCO2017, Hawk FID is 90.71 versus 90.68 for vanilla AR and CLIP score 33.39 versus 33.43 for vanilla AR.
- On Flickr30K, Hawk reaches 1.69× speedup with FID 104.50 versus 104.80 for vanilla AR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine horizontal and vertical draft heads | Hawk on Lumina-mGPT · COCO2017 | Speed up: 1.00× -> 1.71× | Helped |
| combine horizontal and vertical draft heads | Hawk on Lumina-mGPT · Flickr30K | Speed up: 1.00× -> 1.69× | Helped |
| combine horizontal and vertical draft heads instead of horizontal heads alone (ablation) | Hawk on Lumina-mGPT · COCO2017 | Accepted Length: 1.728 -> 1.890 | Helped |
| combine horizontal and vertical draft heads instead of horizontal heads alone (ablation) | Hawk on Lumina-mGPT · Flickr30K | Accepted Length: 1.704 -> 1.871 | Helped |
| use vertical draft heads only instead of horizontal Medusa heads (ablation) | Lumina-mGPT · COCO2017 | Speed up: 1.58× -> 1.58× | No clear effect |
| use vertical draft heads only instead of horizontal Medusa heads (ablation) | Lumina-mGPT · Flickr30K | Speed up: 1.55× -> 1.59× | Helped |
| use relaxed speculative decoding (LANTERN++) instead of Hawk spatial heads | Lumina-mGPT · COCO2017 | FID: 90.71 -> 93.22 | Hurt |
| use relaxed speculative decoding (LANTERN++) instead of Hawk spatial heads | Lumina-mGPT · Flickr30K | FID: 104.50 -> 106.10 | Hurt |
| use relaxed speculative decoding (LANTERN++) instead of Hawk spatial heads | Lumina-mGPT · COCO2017 | CLIP Score: 33.39 -> 33.08 | Hurt |
| use relaxed speculative decoding (LANTERN++) instead of Hawk spatial heads | Lumina-mGPT · Flickr30K | CLIP Score: 34.45 -> 34.25 | Hurt |
| combine horizontal and vertical draft heads instead of horizontal heads alone (ablation) | Hawk on Lumina-mGPT · COCO2017 | FID: 92.51 -> 90.71 | Helped |
| combine horizontal and vertical draft heads instead of horizontal heads alone (ablation) | Hawk on Lumina-mGPT · Flickr30K | FID: 103.51 -> 104.50 | Hurt |
Surprising findings
- Vertical draft heads, despite a much larger speculative distance (one image width), show lower training loss and smaller performance decay with depth than horizontal heads at equal Euclidean distance.
- LANTERN++ achieves higher accepted length than Hawk on some settings in relaxed verification but yields worse FID and CLIP scores.
Limitations
- Speedup is modest (1.71×) and limited by the additional computation of dual-direction draft heads.
- Medusa is used as the base speculative method, which the authors note is not state-of-the-art, so gains may be underestimated.
- Evaluation uses only 500 samples per dataset with a single base model (Lumina-mGPT).
Experimental setup
Datasets: LAION aesthetic COCO 2017 Flickr30K
Benchmarks: COCO 2017 Flickr30K
Metrics: speed up accepted length FID CLIP score
Baselines: Vanilla AR Medusa LANTERN++
Builds on: Medusa speculative sampling tree speculative decoding Lumina-mGPT LANTERN++
Closest papers by meaning
- Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropy (NeurIPS 2025 · similarity 0.92)
- LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding (ICLR 2025 · similarity 0.91)
- ZipAR: Parallel Autoregressive Image Generation through Spatial Locality (ICML 2025 · similarity 0.90)
- Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation (NeurIPS 2025 · similarity 0.90)
- Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding (ICLR 2025 · similarity 0.90)
- Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.