Research map / Architectures and efficiency / Mixture of experts
REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, Vithursan Thangarasa
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prune experts with REAP instead of merging with HC-SMoE | SMoE compression · Qwen3-30B-A3B 50% coding | code average: 0.364 -> 0.541 | Improved |
| prune with REAP instead of merging with M-SMoE | SMoE compression · GLM-4.5-Air 50% math | math average: 0.465 -> 0.857 | Improved |
| use REAP saliency instead of frequency-based pruning (ablation) | SMoE pruning · Qwen3-Coder-480B 50% coding | code average: 0.010 -> 0.619 | Improved |
| use REAP instead of EAN pruning (ablation) | SMoE pruning · Qwen3-30B 50% coding | code average: 0.530 -> 0.541 | Improved |
| use REAP instead of EAN pruning (ablation) | SMoE pruning · Kimi-K2 50% coding | code average: 0.487 -> 0.624 | Improved |
| use REAP instead of EAN pruning (ablation) | SMoE pruning · Kimi-K2 25% tool use BFCLv3 | overall accuracy: 0.652 -> 0.635 | Hurt |
| use merging instead of pruning | SMoE compression · Qwen3-30B 50% multiple-choice average | MC accuracy: 0.503 -> 0.542 | Improved |
| calibrate on C4 instead of domain-specific code data (ablation) | SMoE compression · code generation | accuracy | Hurt |
| restrict maximum merge cluster size to 32 (ablation) | HC-SMoE · code generation | accuracy | Hurt |
| use REAP instead of EAN on Qwen3-30B math at 50% (ablation) | SMoE pruning · Qwen3-30B 50% math | math average: 0.864 -> 0.857 | No clear effect |
Closest papers by meaning
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.91)
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning (NeurIPS 2025 · similarity 0.90)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.90)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.89)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.89)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
