Research map / Large language models / Reasoning and chain of thought

Entropy-Aware On-Policy Distillation of Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EOPD, which adds a forward-KL term on top-k tokens only at high-teacher-entropy positions to on-policy distillation, improving diversity and Pass@8 on six math benchmarks for Qwen3 students.

Problem

On-policy distillation with reverse KL is mode-seeking, which reduces student generation diversity and gives unstable learning signals at high-entropy teacher positions.

Approach

Augment the per-token reverse-KL on-policy distillation loss with a top-k forward KL term applied only where the teacher's token entropy exceeds a threshold (tau=0.8), combining mode-seeking precision with mode-covering coverage of uncertain steps at no extra teacher passes.

Key results

• EOPD improves over OPD by +1.16 Avg@8 and +1.37 Pass@8 averaged over six math benchmarks for Qwen3-0.6B-Base.
• For Qwen3-4B-Base, EOPD improves over OPD by +1.80 Avg@8 and +5.05 Pass@8 on average.
• Standard OPD retains only 6.8% of high-entropy (>=1.0) tokens versus 18.5% for the teacher; EOPD retains substantially more in that region.

Surprising findings

• Advantage shaping drives training entropy to a lower regime and performs worse than OPD, while an entropy bonus maintains entropy comparable to EOPD, so entropy level alone does not explain EOPD's gains.
• Random forward KL on 20% of positions underperforms both full forward KL and EOPD's entropy-selected placement on most benchmarks.

Limitations

• Gains on AIME and AMC are small in absolute terms and based on few problems with Avg@8 over 8 samples.
• Evaluated only on math-focused training with Qwen3 and Llama students; hyperparameters (tau, alpha, k) tuned on the same benchmarks reported.

Experimental setup

Datasets: MATH DAPO-Math-14k

Benchmarks: MATH500 AMC23 Minerva OlympiadBench AIME24 AIME25 GPQA-Diamond MMLU-Pro AlpacaEval 2.0

Metrics: Avg@8 Pass@8 Pass@1 LC-WR WR token-level entropy histogram forward KL at high-entropy tokens

Baselines: KD GRPO OPD GKD OPD + Entropy Bonus OPD + Advantage Shaping BD-KD AKL ToDi

Builds on: on-policy distillation GKD PPO GRPO DistillKit

Closest papers by meaning

• KL for a KL: On-Policy Distillation with Control Variate Baseline (NeurIPS 2026 · similarity 0.92)
• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.91)
• Hybrid Policy Distillation for LLMs (ICML 2026 · similarity 0.91)
• SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.90)
• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.90)
• Learning from Language Feedback via Variational Policy Distillation (NeurIPS 2026 · similarity 0.90)