Research map / Label-efficient and robust learning / Knowledge distillation

When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PW-OPSD, which adds an increasing sigmoid position weight over normalized rollout position to on-policy self-distillation (OPSD), motivated by a branch-viability diagnostic showing teacher-token reliability is positionally structured on Qwen3-4B.

Problem

Standard on-policy self-distillation weights all student-rollout tokens equally, implicitly assuming the privileged teacher target is equally reliable at every prefix; entropy-based adaptive distillation methods conflate high teacher entropy (non-viable uncertainty) with benign diversity.

Approach

A branch-viability diagnostic (forcing teacher-proposed alternatives and testing student-template continuations) shows that teacher unreliability concentrates early in the trace, so PW-OPSD reweights token losses by an increasing sigmoid of normalized position with a per-sequence mean, adding no extra teacher pass.

Key results

• PW-OPSD Moderate improves AIME 2024 Avg@12 from 75.19 (OPSD) to 76.20 and AIME 2025 from 66.67 to 67.78 on Qwen3-4B.
• Position score reaches residual AUROC 0.83 (CI [0.66, 0.95]) for predicting real-uncertain teacher branches, versus at most 0.57 for local uncertainty scores.
• Moderate schedule gives Avg@12 gains across averaged benchmarks on Qwen3-4B (+0.39), DSR1-L8B (+0.35), and Olmo-3-7B-Think (+0.50).

Surprising findings

• Local uncertainty scores (predictive entropy, MC-dropout MI, Dirichlet precision) are near chance for predicting teacher-branch reliability, suggesting entropy-based gating conflates non-viable uncertainty with benign diversity.
• EOPD and REOPOLD regress below OPSD on most benchmarks despite being adaptive methods.

Limitations

• Gains over OPSD are modest (about +1 point on AIME benchmarks) and not uniform across benchmarks; HMMT Moderate trails OPSD on Avg@12.
• Branch-viability diagnostic relies on only 8 real-uncertain vs 271 diversity candidates from 61 problems, a small label set.
• Position schedule hyperparameters were chosen a priori but the sweep shows sensitivity; no method-specific tuning across benchmarks.

Experimental setup

Datasets: OpenThoughts-Math-30k MATH-500 AIME 2024 AIME 2025 HMMT 2025

Benchmarks: MATH-500 AIME 2024 AIME 2025 HMMT 2025

Metrics: Avg@12 Pass@12 Maj@12 AUROC

Baselines: OPSD EOPD REOPOLD

Builds on: OPSD EOPD REOPOLD MiniLLM LoRA

Closest papers by meaning

• Your Teacher Can’t Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation (NeurIPS 2026 · similarity 0.91)
• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.91)
• SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.91)
• SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation (NeurIPS 2026 · similarity 0.89)
• Entropy-Aware On-Policy Distillation of Language Models (ICML 2026 · similarity 0.89)
• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.89)