Research map / Large language models / Reasoning and chain of thought

Speculative Speculative Decoding

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces speculative speculative decoding (SSD), which pre-computes speculations for predicted verification outcomes on a separate draft device during verification, and instantiates it as the Saguaro algorithm with geometric fan-out cache, cache-aware sampling, and batch-size-dependent fallback.

Problem

Speculative decoding still requires drafting to wait for verification to finish, so the sequential dependence between speculation and verification limits decoding latency.

Approach

While the verifier checks round T, a draft model on separate hardware predicts likely verification outcomes (accepted length and bonus token) and pre-speculates for them in a cache; if the actual outcome is cached, the speculation is returned immediately. Fan-out is allocated geometrically over lookahead positions, a cache-aware sampler biases the residual toward cached tokens, and the fallback sp

Key results

• SSD (Saguaro) is on average 30% faster than the strongest speculative decoding baselines (SGLang-based SD, EAGLE-3 where supported).
• Up to 5x faster than autoregressive decoding, with nearly 5x average speedup over the AR baseline on Llama-3.1-70B across four datasets at batch size 1.
• Bonus-token prediction from draft logits reaches up to 90% accuracy.

Surprising findings

• Speculative decoding is generally ineffective for throughput-bound workloads, yet SSD improves both latency and throughput Pareto frontier even counting the extra draft hardware.
• At large batch sizes, a fast random-token backup speculator outperforms a slower accurate neural speculator used just-in-time.

Limitations

• Gains diminish at large batch sizes, where cache misses become frequent and compute becomes the bottleneck.
• Requires a separate draft device, adding hardware cost.
• Speculative decoding generally ineffective for throughput-bound workloads such as large-scale RL or offline data generation.

Experimental setup

Datasets: Alpaca GSM8k UltraFeedback HumanEval

Benchmarks: Alpaca GSM8k UltraFeedback HumanEval

Metrics: end-to-end decoding speedup cache hit rate acceptance rate throughput latency

Baselines: speculative decoding (SD) SGLang vLLM EAGLE-3 autoregressive decoding SwiftSpec SpecBranch

Builds on: speculative decoding EAGLE PEARL AMUSD SpecBranch

Closest papers by meaning

• A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.92)
• SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.92)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.92)
• Block Verification Accelerates Speculative Decoding (ICLR 2025 · similarity 0.92)
• PEARL: Parallel Speculative Decoding with Adaptive Draft Length (ICLR 2025 · similarity 0.91)
• DFlash: Block Diffusion for Flash Speculative Decoding (ICML 2026 · similarity 0.91)