Research map / Large language models / Reasoning and chain of thought

Hybrid Policy Distillation for LLMs

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Hybrid Policy Distillation (HPD), a token-level reweighted log-likelihood KD method combining forward/reverse KL via a masked negative K1 signal on expert tokens and one student-sampled token, achieving SFT-level cost with competitive or better results than SFT, RKLD, JSD, and on-policy distillation.

Problem

Knowledge distillation of LLMs is sensitive to divergence direction, optimization strategy (loss vs reward), and data regime (on- vs off-policy); existing methods either lack mode coverage, are unstable, or are costly due to full-vocabulary matching or full-sequence on-policy rollouts.

Approach

Unifies SFT, forward KL, and reverse KL as token-level reweighted log-likelihood, then defines a hybrid token weight from a negative reverse-K1 gap on the expert token (forward-KL-like when underestimated, masked when overestimated) plus a masked weight on one student-sampled non-expert token obtained during prefilling, with reinforcement of the expert token when the student overestimates a sample

Key results

• On Qwen2.5 7B→1.5B off-policy reasoning, HPD averages 30.24 vs 26.43 for JSD and 22.80 for SFT across six benchmarks; with 3B student the average reaches 39.83 vs 36.61 for JSD.
• On coding, HPD achieves 66.35 average for DS-Coder 1.3B and 77.35 for Qwen-Coder 1.5B, versus 64.10 and 76.00 for JSD.
• Stronger-teacher distillation from Qwen3-30B-A3B-Thinking-2507 into Qwen3-0.6B raises AIME-24 at 38K generation length from 11.35 to 15.78 (+39.03%).

Surprising findings

• Hybrid masked weighting on one student-sampled token outperforms full-distribution KD methods at a fraction of the memory and compute cost in the reported setup.
• On LLaMA 3 1B and coding DS-Coder, HPD does not win on every individual benchmark (e.g., MBPP for DS-Coder where KD scores higher).

Limitations

• Requires teacher and student to share the same tokenizer.
• Gains on some individual benchmarks are not uniform (e.g., MBPP on DS-Coder, LLaMA 3 1B results mixed).
• Full on-policy rollouts are not used, so the method is an initialization rather than a replacement for OPD in some settings.

Experimental setup

Datasets: OpenR1-Math-8192 DAPO-MATH-17k UltraFeedback WizardCoder AlpacaEval MT-Bench Arena-Hard

Benchmarks: AIME24 AIME25 AMC MATH-500 OlympiadBench GPQA AlpacaEval 2 Arena-Hard MT-Bench HumanEval MBPP

Metrics: avg@32 accuracy avg@8 accuracy length-controlled win rate win rate MT-Bench score pass@1 t-test significance

Baselines: SFT SeqKD RKLD JSD KD MiniLLM GKD AKL ABKD

Builds on: MiniLLM GKD SeqKD DPO DAPO

Closest papers by meaning

• KL for a KL: On-Policy Distillation with Control Variate Baseline (NeurIPS 2026 · similarity 0.92)
• Entropy-Aware On-Policy Distillation of Language Models (ICML 2026 · similarity 0.91)
• DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.91)
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.90)
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (ICLR 2024 · similarity 0.90)
• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation (NeurIPS 2026 · similarity 0.90)