Research map / Label-efficient and robust learning / Knowledge distillation
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Speculative Knowledge Distillation (SKD), which interleaves student-proposed and teacher-resampled tokens (accepting a student token only if it is in the teacher's top-K) to train small LLMs on on-policy yet teacher-plausible samples.
Problem
Supervised KD suffers from train-inference distribution mismatch, while on-policy KD trains on low-quality student samples that are OOD for the teacher, and is sensitive to student initialization.
Approach
Student proposes tokens; the teacher checks whether each is within its top-K and, if not, resamples that token from its own distribution (interleaved sampling inspired by speculative decoding). This yields training samples that are on-policy for the student yet likely under the teacher, and SKD interpolates between supervised KD (K small) and on-policy KD (K large).
Key results
- Gemma-7B to Gemma-2B: SKD reaches COMET 75.3 vs 72.5 for SFT and 36.1 for on-policy KD on translation, and GSM8K accuracy 29.1 vs 18.7 SFT and 25.3 on-policy.
- Qwen2-7B to Qwen2-0.5B: SKD reaches ROUGE-L 31.7 on summarization and 36.6 GSM8K accuracy, outperforming supervised KD (31.3, 35.0) in the instruction-tuned-student setting.
- With 10k math prompts, SKD achieves 50.4 SVAMP and 68.4 ASDiv accuracy versus 46.7 and 63.3 for supervised KD (student initialized from Gemma-2B-it).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SKD instead of ImitKD | Gemma-2B-it · Assamese-to-English translation (Flores-200 test) | COMET: 74.8 -> 75.3 | Helped |
| Use SKD instead of ImitKD | Gemma-2B-it · GSM8K test | accuracy: 26.2 -> 29.1 | Helped |
| Use SKD instead of SFT | Qwen2-0.5B-it · Assamese-to-English translation (Flores-200 test) | COMET: 57.6 -> 57.1 | Hurt |
| Use SKD instead of Supervised KD | Qwen2-0.5B-it · GSM8K test | accuracy: 35.0 -> 36.6 | Helped |
| Use SKD instead of On-policy KD | SFT Gemma-2B-it · Dialogue summarization (DialogSum test) | ROUGE-L: 34.8 -> 35.4 | Helped |
| Use SKD instead of Supervised KD | Gemma-2B-it; task-agnostic training with 1k prompts · Math test | accuracy: 7.92 -> 9.56 | Helped |
| Use SKD instead of SFT | Gemma-2B-it; task-agnostic training with 10k prompts · Math test | accuracy: 10.9 -> 10.2 | Hurt |
| Use SKD instead of supervised KD for the first half, then on-policy KD for the second half (ablation) | Student models across three task-specific tasks and two initializations · Task-specific translation, summarization, and arithmetic reasoning | performance | Helped |
| Vary constant top-K from extreme values to a range of 5 to 50 (ablation) | SKD with Gemma-2B-it · Three task-specific text generation tasks | performance | Mixed |
| Decrease K linearly during decoding instead of keeping K=25 (ablation) | SKD · Assamese-to-English translation | COMET: 75.3 -> 75.1; 75.3 -> 75.0 | Hurt |
| Decrease K linearly during decoding instead of keeping K=25 (ablation) | SKD · Dialogue summarization | ROUGE-L: 29.1 -> 25.2; 29.1 -> 28.7 | Hurt |
Surprising findings
- On-policy KD degrades with an instruction-tuned student initialization on translation (COMET 33.7 with IT Gemma-2B in low-data setting), while SKD remains robust.
- Naively concatenating supervised KD and on-policy KD (two-stage) often underperforms either component alone.
Limitations
- Top-K acceptance threshold K=25 is fixed heuristically; adaptive K schedules did not beat constant K.
- Results mostly on 7B-to-2B/0.5B distillation in two model families and a few tasks; SKD's gains are not uniform (e.g., SKD does not beat supervised KD on Qwen translation at 1k).
- Requires teacher and student to share vocabulary for token-level divergence and token acceptance.
Experimental setup
Datasets: Flores-200 DialogSum GSM8K UltraInteract MathQA GSM-plus MATH ASDiv SVAMP
Benchmarks: Flores-200 Assamese-to-English DialogSum GSM8K MATH GSM-plus SVAMP ASDiv
Metrics: COMET-22 ROUGE-L accuracy
Baselines: SFT Supervised KD On-policy KD ImitKD SeqKD
Builds on: on-policy KD speculative decoding ImitKD top-k sampling MiniLLM
Closest papers by meaning
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.91)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.91)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.90)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.89)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.