Research map / Label-efficient and robust learning / Knowledge distillation

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirical study of when on-policy distillation (OPD) and on-policy self-distillation (OPSD) succeed or fail, identifying three failure mechanisms and proposing stop-gradient TopK KL, RLVR-adapted teachers, and SFT stabilization as fixes.

Problem

On-policy (self-)distillation for LLMs gives mixed results: it works for system-prompt and knowledge internalization but reportedly fails or destabilizes on reasoning tasks. The paper asks when OP(S)D works, when it fails, and why.

Approach

Attributes OP(S)D failures to three mechanisms: prefix-induced teacher-student distribution mismatch, biased gradients of unnormalized TopK reverse KL (the +1 term), and OPSD learning a PI-free geometric-mean policy that fails when PI is instance-specific. Proposes stop-gradient TopK KL, RLVR-adapted teachers, and SFT-stabilized students as remedies.

Key results

• Unnormalized Top-20 reverse-KL OPD on Qwen3-1.7B with Qwen3-8B teacher initially improves then collapses to repetitive outputs with Math500/AIME accuracy near zero by step 1000.
• On math, OPSD with final-answer or full-response PI does not stably improve Math500, AIME24, or AIME25; response-PI performs worst.
• Qwen3-1.7B-GRPO teacher (RLVR-adapted, comparable to Qwen3-8B on math) yields significantly better OPD student than the Qwen3-8B teacher, showing teacher accuracy alone does not predict OPD effectiveness.

Surprising findings

• A RL-adapted 1.7B teacher with similar benchmark accuracy to Qwen3-8B distills better than Qwen3-8B, suggesting closeness to student distribution matters more than accuracy.
• Under PI conditioning, the OPSD student performs worse than vanilla OPD even though the PI-conditioned teacher is informative; PI increases teacher-student mismatch on math.

Limitations

• Findings based on a limited set of model families and scales; larger-scale settings may differ.
• Math OPSD failure tested only in the tested settings (Qwen3 models, OpenThoughts/DAPO data).

Experimental setup

Datasets: OpenThoughts DAPO-Math-17k CharacterBench EmotionBench Wildguardmix

Benchmarks: Math500 AIME24 AIME25 GPQA-Diamond CharacterBench EmotionBench Wildguardmix

Metrics: accuracy pass@1 response length training reward evaluation score repetition ratio teacher-student overlap

Baselines: GRPO PPO vanilla OPD Qwen3-8B teacher

Builds on: on-policy distillation GRPO DAPO SDPO PPO

Closest papers by meaning

• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation (NeurIPS 2026 · similarity 0.93)
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.92)
• KL for a KL: On-Policy Distillation with Control Variate Baseline (NeurIPS 2026 · similarity 0.92)
• SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.92)
• When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (NeurIPS 2026 · similarity 0.91)
• Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation (NeurIPS 2026 · similarity 0.91)