Research map / Large language models / Reasoning and chain of thought
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lookahead decoding, an exact parallel decoding method that uses Jacobi iteration trajectories to generate and verify n-grams without a draft model, achieving up to 1.8x speedup on MT-Bench and up to 4x with multi-GPU lookahead parallelism.
Problem
Autoregressive LLM decoding is memory-bandwidth bound and sequential, leading to high latency; existing speedups like speculative decoding need a draft model whose acceptance rate limits gains and which does not generalize.
Approach
Track the Jacobi decoding trajectory in a fixed 2D lookahead window to generate disjoint n-grams in parallel, cache them in an n-gram pool, and verify candidates from the pool in a parallel verification branch within the same forward step, all without an auxiliary model; also introduces lookahead parallelism across GPUs.
Key results
- On MT-Bench with LLaMA-2-chat-7B, lookahead decoding with FlashAttention reaches about 1.8x speedup over autoregressive decoding (1.9x vs 1.07x in the FlashAttention setting).
- Lookahead parallelism on 8 GPUs yields up to about 4x latency reduction on ClassEval code completion, while TP and PP baselines slow down to 0.75x-0.82x.
- On CNN/Daily Mail and XSum with sampling, speedups of 1.46x-1.60x are achieved with ROUGE scores matching autoregressive decoding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Lookahead decoding to autoregressive decoding | LLaMA-2-Chat-7B with FlashAttention · MT-Bench | speedup: 1.07x -> 1.9x | Helped |
| Add FlashAttention | Lookahead decoding · End-to-end inference | speedup: 20% | Helped |
| Use Lookahead Parallelism on 8 GPUs | Code completion · Code completion tasks | speedup: 4x | Helped |
| Use tensor or pipeline parallelism instead of single-GPU serving | Distributed inference · Multiple-GPU settings | speedup: 0.75x-0.82x | Hurt |
| Use Lookahead decoding with sampling | LLaMA-2-7B-Chat · CNN/Daily Mail and XSum | Rouge scores: 36.55 -> 36.53; 19.15 -> 19.20 | No clear effect |
| Use a minimal lookahead branch (W=1) instead of balanced branches (ablation) | Lookahead decoding · MT-Bench | speedup: 1.78x vs 1.36x; 1.04x; 1.46x | Hurt |
| Use a tiny verification branch (G=1) instead of balanced branches (ablation) | Lookahead decoding · MT-Bench | speedup: 1.61x vs 1.78x | Hurt |
| Add prompt as reference to the balanced Lookahead configuration (ablation) | Lookahead decoding (N=5, W=15, G=15) · MT-Bench | speedup: 1.78x -> 1.88x | Helped |
| Use greedy sampling instead of temperature 1.0 sampling | Lookahead decoding · CNN/Daily Mail and XSum | speedup: 1.46x -> 1.57x; 1.50x -> 1.60x | Helped |
Surprising findings
- Tensor and pipeline parallelism baselines slow decoding down (0.75x-0.82x) rather than speeding it up in the strong-scaling setting.
- Jacobi decoding alone hardly reduces the number of decoding steps, motivating the n-gram pool design.
Limitations
- Requires extra computation per step, so it can slow down compute-bound settings such as large batch serving.
- Speedup diminishes as per-step FLOPs grow; to get linear step reduction, FLOPs must grow exponentially.
- Sampling-based decoding yields smaller speedups because acceptance rates are lower.
Experimental setup
Datasets: MT-Bench GSM8K HumanEval MBPP ClassEval XSum CNN/Daily Mail
Benchmarks: MT-Bench GSM8K HumanEval MBPP ClassEval XSum CNN/Daily Mail
Metrics: speedup step compression ratio ROUGE-1 ROUGE-2 ROUGE-L throughput
Baselines: HuggingFace greedy search FlashAttention autoregressive decoding prompt lookup decoding tensor parallelism (DeepSpeed) pipeline parallelism (Accelerate)
Builds on: speculative decoding Jacobi decoding SpecInfer FlashAttention prompt lookup decoding
Closest papers by meaning
- AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.91)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.91)
- SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.91)
- Dynamic-Width Speculative Beam Decoding for LLM Inference (AAAI 2025 · similarity 0.91)
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.90)
- Progressive Mixed-Precision Decoding for Efficient LLM Inference (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.