Research map / Large language models / Reasoning and chain of thought
SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpecExec, a deterministic parallel-search speculative decoding method that builds a large draft "cache" tree of most likely continuations, enabling interactive inference of 50B+ LLMs on consumer GPUs with RAM offloading.
Problem
Interactive LLM inference on consumer GPUs with parameter offloading is bandwidth-bound, and existing speculative decoding methods saturate in accepted tokens as the draft token budget grows.
Approach
Deterministically build a large draft tree of the highest cumulative-probability continuations via a parallel (batched) Dijkstra-style shortest-path search with the draft model, then verify the whole tree in one offloaded target pass; the sampling procedure stays equivalent to sequential sampling from the target model, so it works with deterministic trees, unlike standard speculative sampling.
Key results
- With offloaded Llama 2-70B Chat (16-bit) on A100 and a Llama 2-7B draft, SpecExec reaches 3.12 tokens/s (18.7x speedup, 20.6 accepted tokens per iteration) at budget 2048 on OpenAssistant.
- On consumer GPUs with 4-bit GPTQ Llama 2-70B, SpecExec reaches 5.66 tokens/s on RTX 4090 and 3.68 tokens/s on RTX 3090 (budget 256).
- SpecExec outscales SpecInfer as draft budget grows, with SpecInfer's accepted tokens saturating around 8-10 while SpecExec continues to rise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SpecExec instead of SpecInfer | Llama 2-7B / Llama 2-70B · OpenAssistant | speed (tok/s): 1.34 -> 3.12 | Helped |
| Use SpecExec instead of SpecInfer | Llama 2-7B / Llama 2-70B · C4 | speed (tok/s): 1.03 -> 1.97 | Helped |
| Use SpecExec instead of SpecInfer | Llama 2-7B / Llama 2-70B · WikiText-2 | speed (tok/s): 0.77 -> 1.54 | Helped |
| Increase draft token budget | SpecExec compared with SpecInfer · MTBench and C4 | accepted tokens | Helped |
| Use Llama 2-7B as the draft model instead of smaller draft models (ablation) | SpecExec with Llama 2-70B Chat target · OpenAssistant | accepted tokens | Helped |
| Use the unquantized 13B draft model instead of the 7B draft model (ablation) | SpecExec with Llama 2-70B Chat target · OpenAssistant | accepted tokens | Helped |
| Quantize the 7B draft model to 4 bits with GPTQ (ablation) | SpecExec with Llama 2-70B Chat target · OpenAssistant | accepted tokens | No clear effect |
| Replace parallel SSSP tree search with standard beam search (ablation) | SpecExec · OpenAssistant and WikiText-2 | accepted tokens | Hurt |
| Increase draft tree size beyond its speed-optimal threshold (ablation) | SpecExec · MTBench | generation speed | Hurt |
| Apply heavier token-exclusion penalties (ablation) | SpecExec with TinyLlama-1.1B draft and Llama 2-7B target · MTBench | accepted tokens per iteration | Hurt |
| Apply lightweight token-exclusion penalties (ablation) | SpecExec with TinyLlama-1.1B draft and Llama 2-7B target · MTBench | accepted tokens per iteration | No clear effect |
Surprising findings
- Larger draft models (7B) were far more useful than the usual small draft models, because offloaded target forward passes are slow enough to absorb their cost.
- Standard beam search performed poorly as a draft-tree constructor and plateaued with length, unlike the proposed parallel SSSP search.
Limitations
- Speedups depend on offloading bottleneck; the method is less beneficial without offloading (about 2x on A100 in appendix).
- Optimal tree size and implementation vary by hardware, requiring tuning.
- Authors note the system is not the fastest possible local inference implementation.
Experimental setup
Datasets: OpenAssistant (OASST1) MT-Bench C4 WikiText-2
Benchmarks: OpenAssistant MT-Bench C4 WikiText-2
Metrics: generation rate (accepted tokens per target iteration) tokens per second speedup
Baselines: SpecInfer Sequoia standard sequential offloaded decoding
Builds on: speculative sampling SpecInfer Dijkstra's algorithm GPTQ
Closest papers by meaning
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.93)
- Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.93)
- Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding (NeurIPS 2025 · similarity 0.92)
- SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.92)
- SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding (ICML 2026 · similarity 0.92)
- EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.