Research map / Large language models / Reasoning and chain of thought

PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PADD, a four-stage framework that distills a dense 7B math teacher (no router) into pretrained MoE students (Qwen3-30B-A3B, DeepSeek-V2-Lite) via neuron-cluster expert initialization, online adaptive distillation, routing-shift-suppressed GRPO, and reward-augmented load balancing.

Problem

Dense teachers lack explicit routers, so distilling into MoE students leaves routing policies unsupervised (router cold start, capacity gap, path rupture, expert homogenization) under a fixed inference budget.

Approach

Cluster teacher FFN neurons with cardinality-constrained K-Means to initialize student experts and warm them up with frozen routers; then during RL-style training, scale teacher logits by advantage-dependent temperature, down-weight importance ratios on negative-advantage tokens with large routing shift, and add advantage-based terms to expert routing biases.

Key results

• On Qwen family, PADD averages 80.2% across five math benchmarks vs. 77.7% for the GRPO-trained 7B teacher and 71.4% for MoE-Vanilla-GRPO.
• On DeepSeek family, PADD averages 55.2% vs. 58.1% for the teacher and 46.8% for Vanilla-GRPO.
• PR-GRPO reduces router-shift by about 47% relative to Vanilla-GRPO at step 600 (0.18 vs. 0.34).

Surprising findings

• The MoE student (Qwen3-30B-A3B) with PADD surpasses its 7B dense teacher on the math average.
• Online KD with fixed teacher temperature falls below Vanilla-GRPO on MMLU-Pro for the DeepSeek student.

Limitations

• Ablation and main results are reported on a single training run per setting with few seeds; the paper's own claim of 'multiple seeds' is not fully detailed.
• Evaluated only with math-focused training data and two teacher-student pairs.
• Stage IV bias update and reward-augmented load balancing gains are small (0.6-1.5 points).

Experimental setup

Datasets: DeepScaleR

Benchmarks: AIME24 AMC23 MATH500 Minerva OlympiadBench MMLU-Pro MultiPL-E LiveCodeBench v6 HumanEval MBPP

Metrics: Pass@1 accuracy NMI ESI router-shift expert consistency

Baselines: Base Dense-GRPO MoE-Vanilla-GRPO RSPO GSPO Online KD

Builds on: GRPO GSPO RSPO Online KD Dr.GRPO

Closest papers by meaning

• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
• Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
• TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.89)
• Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.89)
• Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.88)
• All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)