Research map / Large language models / Reasoning and chain of thought

Faster Cascades via Speculative Decoding

ICLR 2025 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Unifies cascades and speculative decoding into a single token-level router framework ("speculative cascades"), deriving optimal routers for greedy and sampling settings that trade off quality and latency.

Problem

Cascades (deferral to a larger model on hard inputs) and speculative decoding (parallel verification by a larger model) optimize inference efficiency via different mechanisms, and their benefits have not been combined or compared under one framework.

Approach

Formulates token-level routing as a loss-minimization problem that recovers greedy and sampling speculative decoding as special cases, and proposes speculative cascades that route based on model confidence (max probability or entropy) with a deferral cost c, implemented speculatively and optionally combined with speculative sampling via a target mixture distribution.

Key results

• Provides derivation that greedy speculative decoding corresponds to router r* that routes to the verifier when drafter and verifier argmax disagree.
• Under calibration assumptions, shows optimal speculative cascade router routes to large model when max p_small < max p_large - c*D_TV.
• Shows speculative sampling is recovered as the c=0 special case of the KL-based router.

Surprising findings

• Speculative cascades can match or exceed the larger model's quality, unlike standard speculative decoding which is only guaranteed to match the larger model.

Limitations

• Calibration assumptions for both models are strong and only approximately hold in practice.
• Optimal router requires ground-truth-related quantities that are not directly available at inference time.
• Provided text contains no empirical evaluation.

Experimental setup

Baselines: speculative decoding cascading (deferral rule) lossy speculative decoding BL-SC

Builds on: speculative decoding speculative sampling cascades lossy speculative decoding

Closest papers by meaning

• SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
• Cascade Speculative Drafting for Even Faster LLM Inference (NeurIPS 2024 · similarity 0.91)
• A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.91)
• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.90)
• CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs (NeurIPS 2025 · similarity 0.90)
• DistillSpec: Improving Speculative Decoding via Knowledge Distillation (ICLR 2024 · similarity 0.89)