Research map / Large language models / Reasoning and chain of thought

Rethinking On-Policy Self-Distillation for Thinking Models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that privileged-context on-policy self-distillation degrades thinking models on AIME/HMMT at long rollout budgets, linking the failure to suppression of forking tokens (fork suppression).

Problem

Self-distillation with privileged teacher context (e.g., gold solutions) is assumed to help self-improvement, but its effect on thinking models that rely on long deliberative reasoning at test time is unclear.

Approach

Empirically shows that privileged-context OPD/OPSD reverses gains of thinking models, and diagnoses this via fork-lock analysis and per-token log-ratio sign reversal on self-correction markers ("fork suppression").

Key results

• OPSD lowers avg@16 average for all five OpenThoughts-trained thinking models, with a relative drop of up to 17% (Qwen3-1.7B: 0.372 to 0.308).
• Vanilla OPD with Qwen3-8B teacher improves Qwen3-1.7B from 0.372 to 0.392, while gold-demo OPD reduces it to 0.350.
• Dense privileged context lowers median fork rate for Qwen3-4B thinking from roughly 0.083 to 0.035, and OPSD reduces verification markers from 1.63 to 1.35 per 1k tokens.

Surprising findings

• Privileged-context OPD degrades thinking models even though vanilla OPD with a larger teacher improves the same student under the same short training budget.
• The gold-demo-conditioned teacher produces much shorter rollouts while keeping high pass@k, but the trained student does not retain its pass@k gains.

Limitations

• Fork-suppression mechanism is correlational and not established as the causal driver of accuracy drop.
• Does not propose a method to fix the degradation.
• Evaluated only on verifiable math and Countdown; other tasks like continual learning not tested.

Experimental setup

Datasets: OpenThoughts math 15k subset (Ashkchamp/Openthoughts_math_filtered_30K) jasonrqh/Countdown-CoT-20k

Benchmarks: AIME24 AIME25 HMMT25 Countdown held-out

Metrics: avg@16 accuracy pass@1 unbiased pass@16 fork rate lock rate deliberation marker density

Baselines: vanilla OPD OPD gold demo base model instruct counterparts

Builds on: on-policy distillation on-policy self-distillation (OPSD) SSD fork/lock diagnostic JSD distillation

Closest papers by meaning

• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.90)
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.89)
• Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information (NeurIPS 2026 · similarity 0.88)
• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.88)
• Rethinking Thinking Tokens: LLMs as Improvement Operators (ICML 2026 · similarity 0.88)
• AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals (NeurIPS 2026 · similarity 0.87)