Research map / Large language models / Reasoning and chain of thought
Online Speculative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes online speculative decoding, which continually distills the draft LLM on target-model corrections from live query traffic (with per-domain draft models), raising token acceptance rates and cutting latency 1.4-2.2x.
Problem
Speculative decoding speedup depends on the draft model's token acceptance rate, which drops for diverse inputs and when the draft is much smaller than the target; a static draft model does not adapt to the live query distribution.
Approach
Treat the speculative decoding verification step as a free source of teacher signal: periodically update the draft model online via knowledge distillation on buffered (draft-error, target-distribution) records from served queries, and route queries to topic- or language-specific draft models.
Key results
- Online updating raises token acceptance rate by about 0.1 to 0.65 across datasets, with comparable results to offline distillation.
- Measured speedup of 1.42x to 2.17x across four datasets with Vicuna-33B/TinyLLaMA-1.1B and Vicuna-7B/LLaMA-160M pairs.
- Combining OSD with Medusa-7B raises Spider speedup from 1.34x to 2.01x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill using teacher-sampled tokens and forward KL instead of the original draft | LLaMA-160M draft for Vicuna-7B · Spider, GSM8K, Code-search-Python, Alpaca-Finance | token acceptance rate: 0.28 -> 0.76; 0.58 -> 0.75; 0.38 -> 0.65; 0.57 -> 0.67 | Helped |
| Distill using teacher-sampled tokens and forward KL instead of the original draft | T5-Small draft for FLAN-T5-XL · Spider, GSM8K, Code-search-Python, Alpaca-Finance | token acceptance rate: 0.13 -> 0.78; 0.29 -> 0.62; 0.28 -> 0.81; 0.39 -> 0.63 | Helped |
| Use teacher-generated label fine-tuning instead of teacher-sampled forward-KL distillation (ablation) | LLaMA-160M draft for Vicuna-7B · Spider, GSM8K, Code-search-Python, Alpaca-Finance | token acceptance rate: 0.74 vs 0.76; 0.74 vs 0.75; 0.65 vs 0.65; 0.68 vs 0.67 | Mixed |
| Use student-sampled tokens instead of teacher-sampled tokens (ablation) | Forward-KL draft distillation · Spider, GSM8K, Code-search-Python, Alpaca-Finance | token acceptance rate: 0.76 -> 0.62; 0.75 -> 0.67; 0.65 -> 0.51; 0.67 -> 0.63 | Hurt |
| Use mixed token sampling instead of teacher sampling (ablation) | Forward-KL draft distillation · Spider, GSM8K, Code-search-Python, Alpaca-Finance | token acceptance rate: 0.76 -> 0.70; 0.75 -> 0.73; 0.65 -> 0.61; 0.67 -> 0.65 | Hurt |
| Replace forward KL with reverse KL or JSD (ablation) | Draft-model distillation · Evaluated tasks | token acceptance rate | No clear effect |
| Online-update the draft model rather than keep it static | Speculative decoding · Four evaluated datasets | speedup: 1.42$\times$ to 2.17$\times$ | Helped |
| Use topic-specific draft models instead of one universal draft model (ablation) | Vicuna-7B speculative decoding on Arena conversations · Topics across Chatbot Arena | token acceptance rate: 0.1 to 0.2 | Helped |
| Fine-tune Medusa heads using OSD | Medusa-7B · Spider | speedup: 1.34$\times$ -> 2.01$\times$ | Helped |
| Fine-tune Medusa heads using OSD | Medusa-7B · Chatbot Arena | speedup: 2.03$\times$ -> 2.38$\times$ | Helped |
| Use standard speculative decoding with online updates instead of Medusa-7B (ablation) | Speculative decoding · Chatbot Arena | speedup: 2.03$\times$ -> 1.51$\times$ | Hurt |
Surprising findings
- Standalone Medusa-7B on the Chatbot Arena dataset (2.03x) outperforms draft-model OSD (1.51x) there, though OSD still improves Medusa when combined.
- Forward KL on teacher samples gave the best acceptance, and reverse KL and JSD did not consistently help, so the best distance measure varies by task.
Limitations
- Routing overhead and memory cost grow with the number of topic-specific draft models (about 10% of target size for five drafts).
- Early in a distribution shift OSD starts below the offline distilled model, and it dips at distribution boundaries.
- Evaluated only on small draft/target pairs with a single serving trace and a 10K LMSYS subset.
Experimental setup
Datasets: Spider Gsm8k Code-search-Python Alpaca-finance LMSYS-Chat-1M
Benchmarks: Spider Gsm8k Code-search-Python Alpaca-finance LMSYS-Chat-1M (Chatbot Arena)
Metrics: token acceptance rate speedup latency per token (ms) FLOPs ratio memory loading bytes
Baselines: static distilled draft model (10% data) Original (no speculative decoding) Medusa-7B label fine-tuning (FT) single draft model
Builds on: speculative decoding generalized knowledge distillation (GKD) Medusa
Closest papers by meaning
- DistillSpec: Improving Speculative Decoding via Knowledge Distillation (ICLR 2024 · similarity 0.92)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.92)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.92)
- Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs (ICLR 2026 · similarity 0.92)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.92)
- Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.