Research map / Architectures and efficiency / Mixture of experts

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations

NeurIPS 2025 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EASY-EP, a one-forward-pass domain-specific expert pruning method for large MoE models (DeepSeek-R1, DeepSeek-V3-0324) that scores experts by gating-weighted output norms and token-level representation change, using few-shot demonstrations.

Problem

Large fine-grained MoE models such as DeepSeek-R1 (671B, 256 experts per layer) require huge memory; existing expert pruning and merging methods are either unreliable at identifying key experts, too expensive with many experts, or designed for models with few experts.

Approach

Observes "few-shot expert localization": a handful of in-domain demonstrations reliably identifies a stable sparse expert subset. Scores each expert by the product of gating value and L2 norm of its output, weighted per token by 1 minus the cosine similarity between hidden states before and after routed experts, then keeps top-M experts in a single forward pass.

Key results

• On DeepSeek-R1 with 128 of 256 experts retained (domain-specific), EASY-EP averages 66.55 vs 67.20 for the full model, outperforming gating-score pruning (56.49).
• Under mixed-domain pruning with 128 experts, EASY-EP averages 61.47 on DeepSeek-R1 and 54.92 on DeepSeek-V3-0324 (full 54.08).
• Pruning to 128 experts yields 2.99x throughput and 64 experts yields 4.33x throughput versus the full DeepSeek-R1 model under SGLang (1K input/1K output).

Measured effects reported in the paper

Surprising findings

• Pruning half the experts with EASY-EP matches or exceeds full-model performance on some domain benchmarks, which the authors attribute to removing irrelevant experts.
• Non-reasoning DeepSeek-V3-0324 is more robust to pruning than reasoning DeepSeek-R1, and its AgentBench-OS score even improves after pruning some experts.

Limitations

• Robustness differs between reasoning and non-reasoning models, validated on only two DeepSeek models.
• Pruning without any training/recovery; the authors leave in-domain vs out-of-domain balancing training to future work.
• Out-of-domain performance is typically poorer than mixed-domain pruning.

Experimental setup

Datasets: AIME-2023 GPQA-main LiveCodeBench-V3 AgentBench-OS dev set FinanceIQ dev set USMLE dev set

Benchmarks: AIME-2024 AIME-2025 HMMT-Feb 2025 LiveCodeBench GPQA-Diamond USMLE FinanceIQ AgentBench-OS

Metrics: pass@1 average accuracy throughput

Baselines: Random pruning Frequency pruning Gating score pruning M-SMoE

Builds on: Mixtral M-SMoE NAEE CD-MoE

Closest papers by meaning

• A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.92)
• Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.91)
• Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
• DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning (NeurIPS 2025 · similarity 0.90)
• Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model (ICLR 2025 · similarity 0.90)
• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.89)