Research map / Large language models / Reasoning and chain of thought

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes On-Policy Self-Distillation (OPSD), where one LLM acts as both a student (question only) and a privileged teacher (question plus reference solution), minimizing per-token divergence over student rollouts, and shows gains on Qwen3 math benchmarks.

Problem

On-policy distillation requires a separate, often larger teacher and does not exploit ground-truth reasoning solutions, while RLVR is sparse and expensive and SFT suffers from distribution mismatch.

Approach

A single LLM serves as both teacher and student under different contexts: the teacher sees the reference solution as privileged information, the student sees only the question; the student samples its own rollouts and receives dense full-vocabulary per-token divergence supervision from the teacher, with per-token pointwise KL clipping for stability.

Key results

• On Qwen3-1.7B, OPSD raises the four-benchmark-average from 37.1 (base) to 43.4 vs 37.7 for GRPO and 35.8 for SFT.
• On Qwen3-8B, OPSD averages 64.8 vs 64.0 for GRPO and 61.8 for the base model, using single rollouts of 1024 tokens per problem and 100 training steps.
• Forward KL gives the best divergence objective on AIME25 with Qwen3-1.7B (43.9 at step 50 vs base 36.7).

Surprising findings

• SFT on the same OpenThoughts data degrades performance below the base model across scales, attributed to concise ground-truth style shortening test-time reasoning.
• Increasing student generation length from 1024 to 4096 tokens did not consistently improve performance.

Limitations

• Experiments limited to models up to 8B parameters.
• Does not explicitly verify correctness of generated answers.
• Effectiveness depends on problem difficulty; teacher gives little supervision on problems beyond model capability.

Experimental setup

Datasets: OpenThoughts (math subset)

Benchmarks: AIME24 AIME25 HMMT25

Metrics: Avg@12 pass@8 accuracy

Baselines: SFT GRPO Qwen3-8B base (Instruct) Qwen3-4B base (Instruct) Qwen3-1.7B base (Instruct)

Builds on: on-policy distillation GKD GRPO STaR context distillation

Closest papers by meaning

• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.92)
• KL for a KL: On-Policy Distillation with Control Variate Baseline (NeurIPS 2026 · similarity 0.92)
• Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning (NeurIPS 2026 · similarity 0.91)
• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation (NeurIPS 2026 · similarity 0.91)
• Hybrid Policy Distillation for LLMs (ICML 2026 · similarity 0.90)
• Entropy-Aware On-Policy Distillation of Language Models (ICML 2026 · similarity 0.90)