Research map / Large language models / Reasoning and chain of thought

Cascade Speculative Drafting for Even Faster LLM Inference

NeurIPS 2024 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Cascade Speculative Drafting (CS Drafting), which recursively cascades draft models (vertical) and assigns larger drafts to earlier tokens (horizontal), with a statistical Max-Gram drafter, to speed up exact speculative decoding.

Problem

Speculative decoding's drafting step remains bottlenecked by autoregressive generation from neural draft models and by equal time allocation to draft tokens that have very different acceptance probabilities, limiting LLM inference speedup.

Approach

Combine a vertical cascade (each neural draft model is itself drafted by a smaller model and reviewed by it, recursively ending in a statistical Max-Gram model) with a horizontal cascade (larger drafts for early tokens, smaller drafts for later tokens), preserving the target model's output distribution.

Key results

• On FLAN-T5 with MS-based speedup, CS Drafting (base, small, MaG) reaches 3.43x on GSM8K and 4.32x on MMLU, versus 2.99x and 3.51x for the best speculative decoding setups.
• On Vicuna-7B, CS Drafting reaches 56.72 tokens/s on GSM8K versus 44.31 for speculative decoding and 33.34 for autoregressive; adding tree attention reaches 63.81, above Medusa's 61.87.
• The paper reports up to 81% additional speedup over speculative decoding on MMLU and up to 44% on GSM8K in standardized walltime improvement.

Measured effects reported in the paper

Surprising findings

• With Max-Gram assistance, FLAN-T5-base consistently outperforms FLAN-T5-small as the single neural draft model in CS Drafting, reversing the speculative decoding trend where small drafts win.

Limitations

• Walltime results depend on hardware and are less reproducible; the authors rely on standardized walltime and theory to complement them.
• Theoretical EWIF analysis assumes i.i.d. token acceptance, a simplified model that may not match real acceptance patterns.
• Experiments are limited to FLAN-T5 and Vicuna-7B with greedy decoding; sampling at temperature 1 was not the main setting.

Experimental setup

Datasets: GSM8K MMLU CNN/DailyMail WMT EnDe Wikipedia (bigram distribution for Max-Gram)

Benchmarks: GSM8K MMLU

Metrics: standardized walltime improvement (speedup) walltime tokens/s expected walltime improvement factor

Baselines: Autoregressive decoding speculative decoding Medusa

Builds on: speculative decoding speculative sampling lenience Medusa tree attention

Closest papers by meaning

• Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration (NeurIPS 2024 · similarity 0.93)
• Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning (ICLR 2026 · similarity 0.92)
• Faster Cascades via Speculative Decoding (ICLR 2025 · similarity 0.91)
• SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting (NeurIPS 2026 · similarity 0.91)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.91)
• Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.91)