Research map / Architectures and efficiency / Mixture of experts
DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiEP, a differentiable non-uniform expert pruning method for MoE LLMs that learns intra-layer and inter-layer importance scores, then prunes globally and adds online expert skipping.
Problem
Uniform per-layer expert pruning in MoE LLMs ignores varying expert redundancy across layers, and exhaustive per-layer combinatorial search is computationally infeasible for models with many experts.
Approach
Relaxes discrete expert selection into continuous learnable intra-layer logits (alpha) and inter-layer scalars (beta), optimizes them with a cross-entropy plus reconstruction regularizer, and prunes globally by the product alpha*beta, enabling non-uniform layer-wise sparsity; also adds similarity-aware expert skipping at inference.
Key results
- On Mixtral 8x7B with 50% experts pruned, DiEP averages 59.9 across MMLU, BoolQ, OpenBookQA and RTE versus 54.6 for NAEE, retaining about 92% of full performance (as stated by the authors).
- DiEP reports up to 7.1% improvement over other pruning methods on MMLU for Mixtral 8x7B.
- Pruning Mixtral 8x7B takes 0.23 hours versus 1.31 hours for NAEE, with 1.28x token generation speedup and 48% memory savings when using expert skipping at 50% sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DiEP instead of S-MOE at 25% expert sparsity | Mixtral 8×7B · Mixtral 8×7B, 25% expert sparsity | Average: 61.5 -> 63.8 | Helped |
| use DiEP instead of S-MOE at 50% expert sparsity | Mixtral 8×7B · Mixtral 8×7B, 50% expert sparsity | Average: 55.9 -> 59.9 | Helped |
| use DiEP instead of NAEE at 25% expert sparsity | Mixtral 8×7B-Instruct · Mixtral 8×7B-Instruct, 25% expert sparsity | Average: 63.8 -> 65.8 | Helped |
| use DiEP instead of NAEE at 50% expert sparsity | Mixtral 8×7B-Instruct · Mixtral 8×7B-Instruct, 50% expert sparsity | Average: 60.9 -> 63.0 | Helped |
| use DiEP instead of S-MOE at 25% expert sparsity | Mixtral 8×22B · Mixtral 8×22B, 25% expert sparsity | Average: 65.7 -> 66.4 | Helped |
| use DiEP instead of S-MOE at 50% expert sparsity | Mixtral 8×22B · Mixtral 8×22B, 50% expert sparsity | Average: 61.3 -> 62.9 | Helped |
| add expert skipping (ablation) | Deepseek-MoE-16B · 0% pruning | Avg. Acc: 56.2 -> 55.7 | Hurt |
| add expert skipping (ablation) | Deepseek-MoE-16B · 6.25% pruning | Avg. Acc: 54.8 -> 54.4 | Hurt |
| add expert skipping (ablation) | Deepseek-MoE-16B · 12.5% pruning | Avg. Acc: 54.1 -> 53.6 | Hurt |
| add expert merging (ablation) | DiEP on Mixtral 8×7B · 25% sparsity | Average: 63.8 -> 64.4 | Helped |
| add expert merging (ablation) | DiEP on Mixtral 8×7B · 50% sparsity | Average: 59.9 -> 60.2 | Helped |
| use 128 rather than 32 C4 calibration samples (ablation) | DiEP on Mixtral 8×7B · pruning to 6 experts | Average: 61.1 -> 63.8 | Helped |
Surprising findings
- Shallower MoE layers receive higher importance scores, and the pruning favors retaining more experts in early layers, contrary to the common assumption of uniform redundancy.
Limitations
- Not evaluated on larger MoE models such as DeepSeek V3 or Qwen2.5-Max due to compute constraints.
- Evaluated only on language models; multimodal MoE (e.g., MoE-LLaVA) not explored.
- Convergence analysis relies on standard smoothness assumptions that may not hold for LLM MoE training.
Experimental setup
Datasets: C4
Benchmarks: MMLU BoolQ OpenBookQA RTE ARC-c ARC-e HellaSwag WinoGrande GSM8K
Metrics: accuracy average accuracy speedup GPU memory
Baselines: M-SMoE Expert Trimming NAEE S-SMoE MC-MoE
Builds on: DARTS DiffPruning NAEE S-SMoE CKA
Closest papers by meaning
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.91)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.90)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.90)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.90)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.89)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.