AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Speculative Self-Distillation enables Efficient Knowledge Internalization

Shayan Talaei, Agam Bhatia, Arshia Soltani Moakhar, Jonas Hübotter, Amin Saberi, Azalia Mirhoseini

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes speculative self-distillation for internalizing knowledge absent from pretraining, where a source-conditioned teacher guides a closed-book student; full text was unavailable so details are limited to the abstract.

Problem

Language models need to internalize knowledge absent from pretraining (proprietary corpora, updated facts, specialized skills) efficiently, and existing self-distillation approaches use a source-conditioned teacher and closed-book student.

Approach

Builds on self-distillation where a model conditioned on source material serves as teacher for an unconditioned student; the title suggests a speculative variant, but the abstract is truncated so the specific mechanism is not stated.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace on-policy rollouts with SSD per-token teacher switchingQwen3-4B self-distillation · Company Memosupervised tokens: 48% fewer tokens, accuracy -1.35 ppImproved
replace on-policy rollouts with SSDQwen3-4B self-distillation · Wikipediasupervised tokens: 38% fewer tokens, accuracy +0.08 ppImproved
replace on-policy rollouts with SSDQwen3-4B · Science Q&Aaccuracy (%): 64.6 -> 66.3Improved
replace on-policy rollouts with SSDQwen2.5-7B · Science Q&Aaccuracy (%): 71.0 -> 73.5Improved
replace on-policy rollouts with SSDQwen3-14B · Science Q&Aaccuracy (%): 73.2 -> 75.5Improved
replace on-policy with SSD in forgetting stress testQwen2.5-7B-Instruct · Forgetting test (six general benchmarks)change in Avg.6: -4.47 -> -1.38Improved
replace off-policy SFT/FKL with SSDQwen2.5-7B-Instruct · Forgetting testchange in Avg.6: -6.64 -> -1.38Improved
use on-policy rollouts instead of off-policy teacher rollouts (ablation)Self-distillation Wikipedia · Wikipediafinal accuracy vs token efficiency: off-policy reaches 78% with ~5x fewer tokens; on-policy higher final accuracyMixed
raise switching threshold tau (more student-sampled) (ablation)SSD · Company Memofinal accuracyImproved
raise switching threshold tau (ablation)SSD · Company Memosupervised tokensHurt
use SSD with concurrent teacher/student decoding vs on-policy trainingQwen3-4B · Science Q&A training timeminutes to on-policy plateau: 414 -> 130Improved

Limitations

Experimental setup

Builds on: self-distillation

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy