Research map / Large language models / Reasoning and chain of thought
Your Teacher Can’t Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lookahead Group Reward (LGR), which augments on-policy reverse-KL distillation with a group-normalized reward on the student's top-K tokens based on the teacher's next-step confidence, plus an entropy-triggered tree-attention implementation, improving math and code reasoning over OPD.
Problem
On-policy distillation's teacher supervision degrades as student-generated prefixes lengthen (Supervision Fidelity Decay), weakening the reverse-KL corrective signal and causing drift over long reasoning chains.
Approach
Score the student's top-K candidate tokens by the teacher's peak next-token probability they induce at the following step, group-normalize these scores into a reward added to the reverse-KL advantage, and compute the teacher passes efficiently with entropy-triggered tree attention.
Key results
- For the 7B student, LGR improves average mean@8 by 2.57 points over OPD (9k), reaching 48.35 average.
- On AIME-26 at 39k max generation length, LGR gains +4.92 points over OPD for the 7B student (per abstract).
- For the 1.5B student, LGR averages 33.18 mean@8, +1.61 over OPD (9k), with the largest gains on AIME-26 and LCB_v6.
Surprising findings
- Setting the confidence reward weight gamma to 10 causes a significant accuracy drop, attributed to reward hacking via repetitive token sequences that inflate teacher confidence.
- On Qwen3 reasoning models, OPD training was highly unstable and degraded progressively, which the authors attribute to the student already being converged on the teacher distribution.
Limitations
- Requires white-box access to teacher logits.
- Relies on the teacher's relative ranking of candidate tokens remaining informative for out-of-distribution trajectories and poorly calibrated teachers.
- Uses a fixed K and static entropy threshold tau.
Experimental setup
Datasets: Polaris DeepScaler code dataset
Benchmarks: AIME-24 AIME-25 AIME-26 HMMT-25 HMMT-26 LiveCodeBench v6
Metrics: mean@8 pass@8
Baselines: GRPO OPD OPD (topk) JSD REOPOLD
Builds on: on-policy distillation GRPO MiniLLM REOPOLD SLIME
Closest papers by meaning
- SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.92)
- When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (NeurIPS 2026 · similarity 0.91)
- SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation (NeurIPS 2026 · similarity 0.90)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
- The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.90)
- Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.