Research map / Large language models / Reasoning and chain of thought
Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces distributed speculative inference (DSI), which overlaps drafting and verification via speculation parallelism so that lossless LM inference is provably never slower than speculative or standard autoregressive inference.
Problem
Speculative inference is sequential (draft then verify), so it can be slower than autoregressive decoding when drafters are slow or inaccurate; existing SI methods do not address this limitation.
Approach
Speculation parallelism: launch many target and drafter threads concurrently over a token tree so verification of drafts overlaps with further drafting; rejected branches are terminated. Proven lossless and at least as fast as non-SI and SI in expectation for any drafter, with lookahead tuning to control the number of target servers.
Key results
- DSI achieves 1.29-1.92x speedup over SI across off-the-shelf target/drafter pairs (e.g., Starcoder-15B/168M on HumanEval 1.92x).
- Offline simulations show DSI never slower than SI or non-SI across configurations, and up to 1.6x faster than the faster of SI/non-SI.
- Theory: DSI is lossless and at least as fast as non-SI, and at least as fast as SI in expectation for any drafter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DSI instead of SI | Starcoder-15B with Starcoder-168M drafter · HumanEval | inference speedup vs. SI: 1.92x | Helped |
| use DSI instead of SI | Starcoder-15B with Starcoder-168M drafter · MBPP | inference speedup vs. SI: 1.66x | Helped |
| use DSI instead of SI | Phi3-14B with Phi3-4B drafter · Alpaca | inference speedup vs. SI: 1.60x | Helped |
| use DSI instead of SI | Phi3-14B with Phi3-4B drafter · HumanEval | inference speedup vs. SI: 1.41x | Helped |
| use DSI instead of SI | Phi3-14B with Phi3-4B drafter · CNN-DM | inference speedup vs. SI: 1.39x | Helped |
| use DSI instead of SI | Phi3-14B with Phi3-4B drafter · MBPP | inference speedup vs. SI: 1.37x | Helped |
| use DSI instead of SI | Vicuna-13B with Vicuna-68M drafter · CNN-DM | inference speedup vs. SI: 1.47x | Helped |
| use DSI instead of SI | Vicuna-13B with Vicuna-68M drafter · Alpaca | inference speedup vs. SI: 1.41x | Helped |
| use DSI instead of SI | Vicuna-7B with Vicuna-68M drafter · CNN-DM | inference speedup vs. SI: 1.29x | Helped |
| use DSI instead of SI | Vicuna-7B with Vicuna-68M drafter · Alpaca | inference speedup vs. SI: 1.70x | Helped |
| use DSI instead of SI or non-SI (ablation) | DSI · offline simulated configurations with non-zero acceptance rate | inference speedup: up to 1.6x | Helped |
| use SI instead of non-SI when the drafter is too slow or inaccurate (ablation) | speculative inference · offline simulated configurations | inference latency | Hurt |
Surprising findings
- SI can be slower than non-speculative inference when drafters are slow or inaccurate, a regime where DSI still gives speedup.
Limitations
- Experiments are simulated: real forward passes replaced by wait commands on a single GPU, not run on an actual 8-GPU node.
- Evaluation limited to single-node setups with SP degree at most 7 and models that fit on one GPU.
- Speedups rely on estimated TTFT/TPOT and acceptance rates from separate experiments.
Experimental setup
Datasets: CNN Daily Mail Alpaca MBPP HumanEval
Benchmarks: HumanEval MBPP CNN-DM Alpaca
Metrics: end-to-end latency speedup acceptance rate TTFT/TPOT latency
Baselines: speculative inference (SI) non-speculative autoregressive inference PEARL
Builds on: speculative sampling speculative decoding SpecInfer
Closest papers by meaning
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.90)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.90)
- Attention-Level Speculation (ICML 2025 · similarity 0.89)
- ATTS: Asynchronous Test-Time Scaling via Conformal Prediction (ICLR 2026 · similarity 0.89)
- SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.89)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.