AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Proposing Better Rollouts for On-Policy Distillation

Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou, Mengting Ai, Zihao Li, Jingrui He

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes changing the rollout policy in on-policy distillation rather than always sampling from the current student, to reduce compounding teacher-student mismatch.

Problem

On-policy distillation for language-model post-training usually samples rollouts directly from the current student, which can create a compounding teacher-student mismatch.

Approach

Treats the rollout policy in OPD as a design choice instead of fixed to the current student, proposing alternative rollout policies that better align sampled trajectories with teacher supervision.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
sample rollouts from a learned geometric-mixture proposal policy instead of the current student (MPOPD vs OPD)OPD · Math avg over 8 benchmarksaccuracy (%): 28.46 -> 35.37 (Qwen3-4B-Base); 16.05 -> 17.51 (Qwen3-1.7B-Base)Improved
apply the geometric mixture to the rollout policy instead of to the distillation target (MPOPD vs VETO)on-policy distillation with student-teacher mixture · Math avg, Qwen3-4B-Baseaccuracy (%): 31.06 -> 35.37Improved
use the MPOPD tilted proposal rolloutOPD · Math avg, Qwen3-4B-Baseaverage generation length: 16.33k -> 9.63kImproved
double OPD student update steps per round (control for extra compute) (ablation)OPD · Math avg, Qwen3-4B-Baseaccuracy (%): 28.46 -> 28.59No clear effect
vary tilt strength eta from 0.10 to 0.30 (ablation)MPOPD · Qwen3-4B-Base math subset (AIME26, H25N, Minerva)accuracy (%): avg 26.93 (0.10), 27.69 (0.20), 27.48 (0.30)No clear effect
reduce teacher top-k support from 16 to 4 (ablation)MPOPD · Math avg, Qwen3-4B-Baseaccuracy (%): 35.37 -> 34.14Hurt
increase teacher top-k support beyond 16 (32, 128) (ablation)MPOPD · Math avg, Qwen3-4B-Baseaccuracy (%): 35.37 -> 35.33 (k=32), 35.66 (k=128)No clear effect
use a larger teacher (Qwen3-30B-A3B-Instruct-2507) and larger studentMPOPD vs OPD · Math avg, Qwen3-4B-Base and Qwen3-8B-Base studentsaccuracy gain over OPD (points): +3.18 (4B), +2.48 (8B) over OPD; +2.17, +1.36 over VETOImproved
use MPOPD instead of OPD on knowledge QA and medical reasoningOPD · Knowledge QA avg / Medical avg, Qwen3-4B-Baseaccuracy (%): 63.16 -> 66.54 (knowledge QA); 57.04 -> 59.17 (medical)Improved
MPOPD under a 4K decoding budget matching training length (ablation)MPOPD vs OPD and VETO · Math avg, Acc@4Kaccuracy (%): OPD 21.89, VETO 24.52, MPOPD 26.89Improved
MPOPD reduces post-answer repetition (tokens after first correct answer)OPD · Math avg, correct responsestokens after first correct answer: 12.38K -> 0.34KImproved
MPOPD vs OPD on weaker 1.7B studentOPD · AIME24, Qwen3-1.7B-Baseaccuracy (%): ExOPD 12.92 vs MPOPD 12.50 on AIME24; avg gain only +0.82Mixed

Experimental setup

Baselines: standard on-policy distillation (student rollouts)

Builds on: on-policy distillation

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy