Research map / Large language models / Reasoning and chain of thought
Speculative decoding
Use draft models and acceptance-rate optimization to decode faster without changing outputs.
92 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2023–2026) · an approach in Reasoning and chain of thought, keyword cluster “decoding · speculative · cache”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 2 tracked benchmarks.
12 papers in 2023–24 and 80 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 9.2% to 5.6%.
Open questions the data points to
- GSM8K (speedup): best went from 2.66 (up to 2024) to 4.48 in 2025 — EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025 · 6 citations)
- MT-Bench (speedup): best went from 2.78 (up to 2024) to 4.4 in 2025 — EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025 · 6 citations)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2023 · Speculative Decoding with Big Little Decoder (NeurIPS 2023 · 9 citations)
- 2024 · EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (ICML 2024 · 6 citations)
- 2025 · EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025 · 6 citations)
- 2026 · SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · 1 citation)
Earliest papers here
- Speculative Decoding with Big Little Decoder (NeurIPS 2023 · 9 citations)
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (ICML 2024 · 6 citations)
- SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · 5 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
GSM8K · speedup (higher is better)
6 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 2.66 | Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration (NeurIPS 2024 · 1 citation) |
| 2025 | 4.48 | EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025 · 6 citations) |
MT-Bench · speedup (higher is better)
4 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 2.78 | Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration (NeurIPS 2024 · 1 citation) |
| 2025 | 4.4 | EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025 · 6 citations) |
What has been tried
42 papers here report 445 measured changes (a component added, removed or swapped, compared with a baseline): helped 71%, no clear effect 5%, hurt 22%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| remove the feature prediction constraint | MT-bench · speedup | 3.16x -> 3.82x | Helped | EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025 · 6 citations) |
| use feature-level autoregression instead of token-level autoregression | MT-bench, Vicuna 7B · speedup | 1.5x to 1.9x | Helped | EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (ICML 2024 · 6 citations) |
| Use Llama 2-7B as the draft model instead of smaller draft models | OpenAssistant · accepted tokens | Helped | SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · 5 citations) | |
| Using model-generated data (draft or target generated) for KD instead of fixed ground-truth data (Supervised KD) | block efficiency across datasets · block efficiency | Helped | DistillSpec: Improving Speculative Decoding via Knowledge Distillation (ICLR 2024 · 4 citations) | |
| Remove the horizontal cascade | GSM8K · Walltime (tokens/s) | 56.16 -> 53.55 | Hurt | Cascade Speculative Drafting for Even Faster LLM Inference (NeurIPS 2024 · 4 citations) |
| Use the Sequoia tree instead of independent sequences | CNN Daily Mail, Llama2-13B, temperature 0.6, tree size ≤512 · generated tokens per decoding step | up to 33% more tokens | Helped | Sequoia: Scalable and Robust Speculative Decoding (NeurIPS 2024 · 4 citations) |
| Use a higher dynamic-width threshold (t=0.9 rather than t=0.7) | Llama-2, Spider · speed | Helped | Dynamic-Width Speculative Beam Decoding for LLM Inference (AAAI 2025 · 3 citations) | |
| Increase the acceptance threshold from 3 to 7 | MATH subdataset · accuracy | 59.4% to 63.7% | Helped | SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning (NeurIPS 2025 · 3 citations) |
| Use teacher-generated label fine-tuning instead of teacher-sampled forward-KL distillation | Spider, GSM8K, Code-search-Python, Alpaca-Finance · token acceptance rate | 0.74 vs 0.76; 0.74 vs 0.75; 0.65 vs 0.65; 0.68 vs 0.67 | Mixed | Online Speculative Decoding (ICML 2024 · 3 citations) |
| Use a minimal lookahead branch (W=1) instead of balanced branches | MT-Bench · speedup | 1.78x vs 1.36x; 1.04x; 1.46x | Hurt | Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (ICML 2024 · 3 citations) |
| add the target model's LM Head to the Kangaroo adapter | Spec-Bench average · walltime speedup | 1.50 × -> 1.44 × | Hurt | Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting (NeurIPS 2024 · 2 citations) |
| add tree attention rather than using an equal-height chain | MT-Bench, Vicuna-7B · speedup | 2.65x -> 3.10x | Helped | Falcon: Faster and Parallel Inference of Large Language Models Through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree (AAAI 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 80 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- GSM8K · 37 of 80 papers
- HumanEval · 27 of 80 papers
- ShareGPT · 18 of 80 papers
- MT-Bench · 17 of 80 papers
- Alpaca · 17 of 80 papers
- CNN/Daily Mail · 15 of 80 papers
- Spec-Bench · 8 of 80 papers
- MATH500 · 7 of 80 papers
Baselines
- EAGLE-3 · 22 of 80 papers
- Medusa · 14 of 80 papers
- EAGLE-2 · 13 of 80 papers
- EAGLE · 9 of 80 papers
- speculative decoding · 6 of 80 papers
- SpS · 6 of 80 papers
- PLD · 6 of 80 papers
- Lookahead · 6 of 80 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 92 papers; from OpenAlex and the papers' affiliations).
- Carnegie Mellon University · 8 papers
- Shanghai Jiao Tong University · 4 papers
- University of California, Berkeley · 4 papers
- Meta · 4 papers
- Google · 3 papers
- Princeton University · 3 papers
- Microsoft · 3 papers
- Peking University · 3 papers
- Stanford University · 3 papers
- University of California, San Diego · 3 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient LLM serving (126 papers)
- KV cache compression (45 papers)
Elsewhere in the topic
- RL with verifiable rewards for reasoning (270 papers)
- Efficient and adaptive thinking (231 papers)
- Test-time compute scaling (205 papers)
- Long CoT dynamics and efficiency (136 papers)
- Routing, pruning and distillation for reasoning (135 papers)
- Chain-of-thought training and analysis (127 papers)
- Search and self-correction in reasoning (123 papers)
- Process reward models (39 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).