Research map / Large language models / Reasoning and chain of thought
DistillSpec: Improving Speculative Decoding via Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DistillSpec, which distills a small draft model from a larger target model using on-policy data and tailored divergences so that speculative decoding achieves higher acceptance rates and 10-45% speedups.
Problem
Speculative decoding speedup depends on how well the small draft model's token distribution matches the large target model, and off-the-shelf small models are poorly aligned, limiting acceptance rates and wall-clock gains.
Approach
Use white-box knowledge distillation to align the draft model with the target model specifically for speculative decoding, finding that on-policy (draft-generated) data is sufficient and that the best divergence (FKL, JSD, RKL, TVD) is task- and decoding-strategy dependent; also extends to lossy SD with lenience functions and to combining distillation of the target with SD in a model garden.
Key results
- DistillSpec yields 10-45% decoding speedup over standard speculative decoding across XSum, CNN/DM, WMT, and GSM8K with greedy and non-greedy sampling.
- A draft distilled on GSM8K transfers to 23 BigBenchHard tasks with average speedup rising from 1.93x to 2.21x (greedy) and 1.78x to 2.02x (non-greedy).
- In the model garden setting, DistillSpec reduces relative latency from 17.3 to 2.7 on XSum (6.4x) and 15.0 to 1.4 on GSM8K (10.7x) with near-identical quality (GSM8K accuracy 33.1 to 34.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| White-box KD of draft model using on-policy (draft-generated) data with target logits, versus standard SD | T5 draft models for speculative decoding · XSum, CNN/DM, WMT, GSM8K, LM1B | latency speedup / block efficiency: 10-46% speedup improvement | Helped |
| Using model-generated data (draft or target generated) for KD instead of fixed ground-truth data (Supervised KD) (ablation) | draft model distillation · block efficiency across datasets | block efficiency | Helped |
| Using SeqKD (black-box, teacher-sample only, no teacher logits) instead of white-box distillation (ablation) | draft model distillation · block efficiency | block efficiency | Hurt |
| Using draft-model on-policy data for KD instead of target-generated data (ablation) | draft model distillation (GKD) · XSum, GSM8K | block efficiency / wall-time speedup | No clear effect |
| Directly minimizing TVD as the KD divergence (ablation) | draft distillation for SD · XSum and GSM8K recipe search | block efficiency | No clear effect |
| Combining KD on target with SD using distilled draft (DistillSpec) vs raw target latency | T5 model garden, XSum · XSum Rouge2 and relative latency | relative latency: 6.4x speedup | Helped |
| Combining KD on target with SD using distilled draft (DistillSpec) vs raw target latency | T5 model garden, GSM8K · GSM8K accuracy and relative latency | relative latency: 10.7x speedup; accuracy 33.1 to 34.8 | Helped |
| Distilling draft on GSM8K and transferring to 23 unseen BigBenchHard tasks | T5-Small draft with T5-XL target · BigBenchHard average speedup | average speedup: 1.93x to 2.21x (greedy); 1.78x to 2.02x (non-greedy) | Helped |
| Lossy SD with leniency combined with distilled draft (vs. non-distilled) quality-latency trade-off | GSM8K with lenience functions · GSM8K quality vs latency | quality-latency trade-off | Helped |
| Choice of divergence (FKL, JSD, RKL, TVD) for KD is task- and decoding-dependent (ablation) | draft distillation recipe · XSum greedy, GSM8K greedy and non-greedy | block efficiency | Mixed |
Surprising findings
- Distilled models with high task performance do not necessarily make better draft models; block efficiency gains and task-quality gains are only weakly correlated.
- Directly minimizing TVD, the theoretically motivated objective, does not give the best results in most settings; the best divergence is task- and decoding-dependent.
Limitations
- Gains depend on task and decoding strategy, so divergence and data choices must be tuned per task.
- Distilled models were not fully optimized when paired with mismatched target sizes (T5-XXL target with T5-Small draft from T5-XL distillation).
- Latency measured on a single TPUv4 device with batch size 1; generalization to other hardware or batch sizes not established.
Experimental setup
Datasets: XSum CNN/DM WMT14 EnDe GSM8K LM1B BigBenchHard
Benchmarks: XSum CNN/DM WMT14 EnDe GSM8K LM1B BigBenchHard
Metrics: block efficiency acceptance rate wall-clock speedup ROUGE-2 BLEU accuracy
Baselines: standard speculative decoding SeqKD Supervised KD ImitKD f-Distill GKD
Builds on: speculative sampling generalized knowledge distillation (GKD) f-distill SeqKD lossy speculative decoding
Closest papers by meaning
- AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders (NeurIPS 2025 · similarity 0.94)
- Online Speculative Decoding (ICML 2024 · similarity 0.92)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.92)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.91)
- Flatter Tokens are More Valuable for Speculative Draft Model Training (ICLR 2026 · similarity 0.90)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.