Research map / Label-efficient and robust learning / Knowledge distillation
Proposing Better Rollouts for On-Policy Distillation
Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou, Mengting Ai, Zihao Li, Jingrui He
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes changing the rollout policy in on-policy distillation rather than always sampling from the current student, to reduce compounding teacher-student mismatch.
Problem
On-policy distillation for language-model post-training usually samples rollouts directly from the current student, which can create a compounding teacher-student mismatch.
Approach
Treats the rollout policy in OPD as a design choice instead of fixed to the current student, proposing alternative rollout policies that better align sampled trajectories with teacher supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sample rollouts from a learned geometric-mixture proposal policy instead of the current student (MPOPD vs OPD) | OPD · Math avg over 8 benchmarks | accuracy (%): 28.46 -> 35.37 (Qwen3-4B-Base); 16.05 -> 17.51 (Qwen3-1.7B-Base) | Improved |
| apply the geometric mixture to the rollout policy instead of to the distillation target (MPOPD vs VETO) | on-policy distillation with student-teacher mixture · Math avg, Qwen3-4B-Base | accuracy (%): 31.06 -> 35.37 | Improved |
| use the MPOPD tilted proposal rollout | OPD · Math avg, Qwen3-4B-Base | average generation length: 16.33k -> 9.63k | Improved |
| double OPD student update steps per round (control for extra compute) (ablation) | OPD · Math avg, Qwen3-4B-Base | accuracy (%): 28.46 -> 28.59 | No clear effect |
| vary tilt strength eta from 0.10 to 0.30 (ablation) | MPOPD · Qwen3-4B-Base math subset (AIME26, H25N, Minerva) | accuracy (%): avg 26.93 (0.10), 27.69 (0.20), 27.48 (0.30) | No clear effect |
| reduce teacher top-k support from 16 to 4 (ablation) | MPOPD · Math avg, Qwen3-4B-Base | accuracy (%): 35.37 -> 34.14 | Hurt |
| increase teacher top-k support beyond 16 (32, 128) (ablation) | MPOPD · Math avg, Qwen3-4B-Base | accuracy (%): 35.37 -> 35.33 (k=32), 35.66 (k=128) | No clear effect |
| use a larger teacher (Qwen3-30B-A3B-Instruct-2507) and larger student | MPOPD vs OPD · Math avg, Qwen3-4B-Base and Qwen3-8B-Base students | accuracy gain over OPD (points): +3.18 (4B), +2.48 (8B) over OPD; +2.17, +1.36 over VETO | Improved |
| use MPOPD instead of OPD on knowledge QA and medical reasoning | OPD · Knowledge QA avg / Medical avg, Qwen3-4B-Base | accuracy (%): 63.16 -> 66.54 (knowledge QA); 57.04 -> 59.17 (medical) | Improved |
| MPOPD under a 4K decoding budget matching training length (ablation) | MPOPD vs OPD and VETO · Math avg, Acc@4K | accuracy (%): OPD 21.89, VETO 24.52, MPOPD 26.89 | Improved |
| MPOPD reduces post-answer repetition (tokens after first correct answer) | OPD · Math avg, correct responses | tokens after first correct answer: 12.38K -> 0.34K | Improved |
| MPOPD vs OPD on weaker 1.7B student | OPD · AIME24, Qwen3-1.7B-Base | accuracy (%): ExOPD 12.92 vs MPOPD 12.50 on AIME24; avg gain only +0.82 | Mixed |
Experimental setup
Baselines: standard on-policy distillation (student rollouts)
Builds on: on-policy distillation
Closest papers by meaning
- The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.92)
- AsyncOPD: How Stale Can On-Policy Distillation Be? (NeurIPS 2026 · similarity 0.89)
- Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation (NeurIPS 2026 · similarity 0.89)
- SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation (NeurIPS 2026 · similarity 0.89)
- When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (NeurIPS 2026 · similarity 0.88)
- SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
