Research map / Large language models / Reasoning and chain of thought
AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaSPEC, which uses a reference model distilled from the target to filter out hard-to-fit tokens so the draft model's forward-KL distillation focuses on tokens that matter for speculative decoding acceptance.
Problem
Standard knowledge distillation minimizes KL over all tokens, which is misaligned with speculative decoding's goal of maximizing token acceptance rate; small draft models with limited capacity waste capacity on hard-to-learn tokens.
Approach
A reference model (a copy of the draft, distilled from the target) is used to score tokens by the KL gap between the draft and reference; the top-k fraction of tokens with largest gap is kept and the draft is distilled only on those tokens with forward KL.
Key results
- On Pythia-31M to 1.4B, AdaSPEC improves acceptance rate over DistillSpec on all five tasks in the 3-Epoch setting (e.g., GSM8K 57.58% to 62.63%).
- With CodeGen-350M to Phi-2 under Optimal-Epoch, MBPP acceptance rises from 86.60% to 87.70%, and GSM8K from 81.49% to 83.48%.
- With EAGLE on Vicuna-7B-v1.3, training accuracy rises from 75.3% to 76.3% and tokens/s from 63.48 to 68.21.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdaSPEC selective token filtering instead of DistillSpec | Pythia-31M → Pythia-1.4B and CodeGen-350M → Phi-2 draft models · GSM8K, Alpaca, MBPP, CNN/Daily Mail, and XSUM; 3-epoch and optimal-epoch settings | acceptance rate | Helped |
| Train on top 40% instead of bottom 40% of selected tokens (ablation) | Pythia-31M → Pythia-1.4B draft model · GSM8K | acceptance rate: 49.03% -> 63.22% | Helped |
| Train on top 40% instead of bottom 40% of selected tokens (ablation) | Pythia-31M → Pythia-1.4B draft model · MBPP | acceptance rate: 39.75% -> 48.22% | Helped |
| Replace distillation with direct fine-tuning (ablation) | AdaSPEC token-selected draft model · GSM8K and MBPP | acceptance rate: 63.22% -> 63.13%; 48.22% -> 45.61% | Hurt |
| Replace forward KL with TVD (ablation) | AdaSPEC draft model · GSM8K and MBPP | acceptance rate: 63.22% -> 9.09%; 48.22% -> 6.76% | Hurt |
| Replace forward KL with RKL (ablation) | AdaSPEC draft model · GSM8K and MBPP | acceptance rate: 63.22% -> 30.05%; 48.22% -> 15.61% | Hurt |
| Vary token selection ratio toward lower k values (0.2–0.4) (ablation) | AdaSPEC draft model · GSM8K | acceptance rate | Helped |
| Use AdaSPEC instead of DistillSpec | Pythia-31M → Pythia-1.4B speculative decoding · MBPP | speed (tokens/s): 149.15 -> 181.67 | Helped |
| Use AdaSPEC instead of DistillSpec | Qwen2.5-0.5B → Qwen2.5-32B draft model · GSM8K, 3 epochs | acceptance rate: 84.43% -> 86.21% | Helped |
Surprising findings
- Using the bottom 40% of tokens (highest-loss) yields a draft worse than the reference model, confirming the value of selection direction.
- Applying the TVD or reverse KL objectives with token selection degrades acceptance rates, so the gain is specific to forward KL.
Limitations
- Filtering is a simple loss-based token filter; more adaptive strategies are not explored.
- Gains are modest in several settings (often a few percentage points) and compared mainly against DistillSpec.
- Experiments limited to small Pythia/CodeGen models; scaling to larger models shown only in one Qwen GSM8K experiment.
Experimental setup
Datasets: GSM8K Alpaca MBPP CNN/Daily Mail XSUM ShareGPT
Benchmarks: GSM8K Alpaca MBPP CNN/Daily Mail XSUM MT-Bench
Metrics: acceptance rate block efficiency wall-clock speed (s/sentence, tokens/s) training accuracy
Baselines: DistillSpec EAGLE
Builds on: DistillSpec speculative decoding EAGLE Pythia
Closest papers by meaning
- DistillSpec: Improving Speculative Decoding via Knowledge Distillation (ICLR 2024 · similarity 0.94)
- Online Speculative Decoding (ICML 2024 · similarity 0.91)
- TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification (NeurIPS 2026 · similarity 0.91)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.90)
- GRIFFIN: Effective Token Alignment for Faster Speculative Decoding (NeurIPS 2025 · similarity 0.90)
- Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.