Research map / Large language models / Reasoning and chain of thought

Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes parameter-level mechanisms of on-policy distillation (OPD) versus RL for LLM reasoning, finding OPD concentrates updates in critical modules and early low-rank directions, and proposes EffOPD, an adaptive directional extrapolation that accelerates OPD about 3x.

Problem

The parameter-level mechanisms behind the efficiency of on-policy distillation (OPD) for LLM post-training are poorly understood; existing explanations rely on denser, more stable supervision.

Approach

OPD shows "foresight": it suppresses updates in low-marginal-utility modules and early locks its update directions into a stable low-rank subspace aligned with the final update, so extrapolating along the early direction with validation-gated step sizes (EffOPD) accelerates training.

Key results

• EffOPD achieves an average training acceleration of about 3x over vanilla OPD while maintaining comparable final performance across 1.5B-32B models.
• On the 8B model, OPD shows higher spectral/Frobenius ratio (36.8% vs 32.7% for RL) and lower effective rank (2341 vs 2754) than RL.
• A module-wise norm-scaled OPD checkpoint at 10% training progress recovers about 80% of the final model's reasoning performance.

Surprising findings

• Embedding-layer replacement with base-model embeddings has negligible effect on reasoning performance for OPD and RL.
• RL shows a slight advantage over OPD in tail (bottom-k%) subspaces, though at 1.6-2.5x the norm cost.

Limitations

• Applicability to multi-turn agent tasks and multimodal reasoning is untested.
• Theoretical analysis is local, around the base model, and does not capture global non-convex behavior.
• Evaluation focuses on math and code reasoning with Qwen-family models.

Experimental setup

Datasets: DeepMath-103K MATH-12K Eurus-RL-Code

Benchmarks: Codeforces TACO AIME24 AIME25 AIME26 MINERVA GPQA MATH500

Metrics: accuracy training speedup spectral norm effective rank subspace alignment

Baselines: Vanilla OPD AlphaOPD ExOPD PPO GRPO DAPO

Builds on: on-policy distillation AlphaRL GRPO DAPO PPO

Closest papers by meaning

• KL for a KL: On-Policy Distillation with Control Variate Baseline (NeurIPS 2026 · similarity 0.93)
• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.93)
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.91)
• AsyncOPD: How Stale Can On-Policy Distillation Be? (NeurIPS 2026 · similarity 0.90)
• Hybrid Policy Distillation for LLMs (ICML 2026 · similarity 0.90)
• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.89)