Research map / Label-efficient and robust learning / Knowledge distillation
Speculative Self-Distillation enables Efficient Knowledge Internalization
Shayan Talaei, Agam Bhatia, Arshia Soltani Moakhar, Jonas Hübotter, Amin Saberi, Azalia Mirhoseini
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes speculative self-distillation for internalizing knowledge absent from pretraining, where a source-conditioned teacher guides a closed-book student; full text was unavailable so details are limited to the abstract.
Problem
Language models need to internalize knowledge absent from pretraining (proprietary corpora, updated facts, specialized skills) efficiently, and existing self-distillation approaches use a source-conditioned teacher and closed-book student.
Approach
Builds on self-distillation where a model conditioned on source material serves as teacher for an unconditioned student; the title suggests a speculative variant, but the abstract is truncated so the specific mechanism is not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace on-policy rollouts with SSD per-token teacher switching | Qwen3-4B self-distillation · Company Memo | supervised tokens: 48% fewer tokens, accuracy -1.35 pp | Improved |
| replace on-policy rollouts with SSD | Qwen3-4B self-distillation · Wikipedia | supervised tokens: 38% fewer tokens, accuracy +0.08 pp | Improved |
| replace on-policy rollouts with SSD | Qwen3-4B · Science Q&A | accuracy (%): 64.6 -> 66.3 | Improved |
| replace on-policy rollouts with SSD | Qwen2.5-7B · Science Q&A | accuracy (%): 71.0 -> 73.5 | Improved |
| replace on-policy rollouts with SSD | Qwen3-14B · Science Q&A | accuracy (%): 73.2 -> 75.5 | Improved |
| replace on-policy with SSD in forgetting stress test | Qwen2.5-7B-Instruct · Forgetting test (six general benchmarks) | change in Avg.6: -4.47 -> -1.38 | Improved |
| replace off-policy SFT/FKL with SSD | Qwen2.5-7B-Instruct · Forgetting test | change in Avg.6: -6.64 -> -1.38 | Improved |
| use on-policy rollouts instead of off-policy teacher rollouts (ablation) | Self-distillation Wikipedia · Wikipedia | final accuracy vs token efficiency: off-policy reaches 78% with ~5x fewer tokens; on-policy higher final accuracy | Mixed |
| raise switching threshold tau (more student-sampled) (ablation) | SSD · Company Memo | final accuracy | Improved |
| raise switching threshold tau (ablation) | SSD · Company Memo | supervised tokens | Hurt |
| use SSD with concurrent teacher/student decoding vs on-policy training | Qwen3-4B · Science Q&A training time | minutes to on-policy plateau: 414 -> 130 | Improved |
Limitations
- Full text unavailable; abstract is truncated
Experimental setup
Builds on: self-distillation
Closest papers by meaning
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.90)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.90)
- The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.89)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.88)
- SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.88)
- When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
