Research map / Architectures and efficiency / Mixture of experts

Pruning and Distilling Mixture-of-Experts into Dense Language Models

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Converts a trained MoE (Qwen3-30B-A3B) into a fully dense 3.3B FFN student by scoring, selecting/merging experts (with a D-optimal Gram log-determinant selection, DO-ACP) and distilling with forward KL, outperforming dense-to-dense pruning.

Problem

MoE models must keep all experts in memory, which hinders memory-constrained deployment; prior MoE compression still outputs MoE models, and no work has systematically converted MoE into fully dense models.

Approach

Score experts, select a top-K subset using a diversity-aware D-optimal (Gram log-determinant) criterion on importance-weighted expert output Gram matrices, optionally merge into k groups, concatenate into one dense FFN with down-projection scaling, then recover quality by forward-KL distillation from the MoE teacher.

Key results

• DO-ACP at K=8 reaches 58.10% average accuracy after ~4B-token distillation, +6.3 pp over dense-to-dense pruning (51.84%) from Qwen3-32B.
• Scoring is the dominant factor: best (DO-ACP, 42.64% mean) vs worst scoring family (PP, 36.94%) differ by 5.7 pp, while grouping spread is ~1 pp.
• MoE-to-dense distillation runs at 73 s/step vs 116 s/step for dense teacher (1.6x speedup) on 2xB200 GPUs.

Surprising findings

• Pure pruning (no weight averaging) beats merging for diversity-aware scoring, even though merging yields lower pre-distillation perplexity.
• A dense-to-dense pruned student from a matched-size dense teacher is barely above a random-FFN baseline (33.28% vs 32.70%).

Limitations

• Merging ties merge weights to selection scores, which may limit merged configurations.
• Extended training only reaches ~4B tokens; quality ceiling at larger scale is unknown.
• Benefit over random FFN initialization shrinks for GPT-OSS with only 32 experts.

Experimental setup

Datasets: WikiText-103 WikiText-2 FineWeb-Edu MMLU Winogrande HellaSwag ARC-Easy ARC-Challenge

Benchmarks: Winogrande HellaSwag ARC-Easy ARC-Challenge MMLU WikiText-2

Metrics: accuracy average accuracy perplexity validation CE loss wall-clock time per step

Baselines: MC-SMoE HC-SMoE dense-to-dense pruning (Minitron-style) random FFN + teacher attention random initialization SF REAP

Builds on: REAP Minitron MC-SMoE HC-SMoE SwiGLU

Closest papers by meaning

• Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.94)
• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.94)
• Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.93)
• BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.93)
• Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · similarity 0.92)
• LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.92)