Research map / Image generation / Autoregressive and latent generation
LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies speculative decoding to visual autoregressive image generation and proposes LANTERN, a relaxed acceptance condition that aggregates probabilities over latent-space nearest-neighbor tokens, bounded by a total variation distance, achieving up to 2.26x speed-up on LlamaGen.
Problem
Speculative decoding gives poor speed-ups when naively applied to visual autoregressive image models because their next-token distributions are near-uniform (token selection ambiguity), so drafted tokens are frequently rejected.
Approach
Relax the speculative acceptance test by summing target probability over a set of latent-space nearest-neighbor tokens of the draft token (exploiting token interchangeability), and cap the resulting distribution distortion with a total variation distance bound that determines the neighborhood size.
Key results
- On LlamaGen-XL Stage I with greedy decoding, LANTERN (delta=0.2, k=1000) reaches 2.26x speed-up (MAL 2.89) vs 1.29x for EAGLE-2, with FID rising from 28.63 to 30.78.
- With sampling (tau=1), LANTERN (delta=0.4, k=1000) achieves 1.69x speed-up vs 0.93x for EAGLE-2, with FID 18.76 vs 15.22 for vanilla AR.
- On Anole, LANTERN gives 1.17x speed-up versus 0.73x for EAGLE-2 under sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Relax speculative acceptance using latent neighbors with TVD bound δ=0.2 rather than standard decoding | LlamaGen-XL Stage I · MS-COCO validation, greedy (τ=0) | speed-up: 1.00× -> 2.26× | Helped |
| Relax speculative acceptance using latent neighbors with TVD bound δ=0.4 rather than standard decoding | LlamaGen-XL Stage I · MS-COCO validation, sampling (τ=1) | speed-up: 1.00× -> 1.69× | Helped |
| Increase the TVD bound from δ=0.05 to δ=0.2 | LANTERN on LlamaGen-XL Stage I · MS-COCO validation, greedy (τ=0) | speed-up: 1.56× -> 2.26× | Helped |
| Increase the TVD bound from δ=0.1 to δ=0.4 | LANTERN on LlamaGen-XL Stage I · MS-COCO validation, sampling (τ=1) | speed-up: 1.13× -> 1.69× | Helped |
| Use latent-neighbor selection instead of random selection (ablation) | LANTERN on LlamaGen-XL Stage I · MS-COCO validation, sampling (τ=1) | mean accepted length: 1.26 -> 2.09 | Helped |
| Use cosine similarity rather than ℓ2 distance to select latent neighbors (ablation) | LANTERN on LlamaGen-XL Stage I · MS-COCO validation, sampling (τ=1) | mean accepted length: 2.09 -> 2.09 | No clear effect |
| Set δ=0.05 instead of δ=0.2 (ablation) | LANTERN with ℓ2 latent-neighbor selection · MS-COCO validation, sampling (τ=1) | mean accepted length: 2.09 -> 1.50 | Hurt |
| Use JSD rather than TVD to constrain distributional divergence (ablation) | LANTERN on LlamaGen-XL Stage I · MS-COCO validation, sampling (τ=1), matched mean accepted length | FID: 17.43 -> 17.48 | No clear effect |
Surprising findings
- The EAGLE-2 drafter reaches an average accept probability of only 0.0402 on visual AR models, meaning about 4% of drafts are accepted.
- Naive speculative decoding methods (EAGLE-2) yield speed-ups below 1x under sampling, i.e., slower than vanilla decoding.
Limitations
- Image quality degrades somewhat (FID rises, recall drops) as speed-up increases.
- Relies on a drafter trained per visual AR model; the drafter accuracy remains low, and the paper leaves a visual-specific drafter design to future work.
- Proximity set computation adds overhead, growing at lower delta values.
Experimental setup
Datasets: LAION-COCO ImageNet MS-COCO 2017
Benchmarks: MS-COCO validation captions
Metrics: speed-up mean accepted length FID CLIP score Precision Recall HPS v2
Baselines: Vanilla AR EAGLE-2 Medusa Speculative Decoding
Builds on: speculative decoding EAGLE-2 Medusa DistillSpec
Closest papers by meaning
- Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient (CVPR 2025 · similarity 0.91)
- Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation (NeurIPS 2025 · similarity 0.91)
- Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding (ICLR 2025 · similarity 0.90)
- Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching (ICLR 2025 · similarity 0.89)
- Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation (NeurIPS 2025 · similarity 0.89)
- RandAR: Decoder-only Autoregressive Visual Generation in Random Orders (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.