Research map / Architectures and efficiency / Mixture of experts
Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
Wentao Hu, Mingkuan Zhao, Shuangyong Song, Xiaoyan Zhu, Xin Lai, Jiayin Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add global activation-variability selection (GVP) over reconstruction-only Enumeration (ablation) | Enumeration Pruning · 7 general benchmarks (Mixtral, 6 experts) | average accuracy: 72.79 -> 73.50 | Improved |
| add cluster-then-select on top of GVP (full MoP) (ablation) | GVP · 7 general benchmarks (Mixtral, 6 experts) | average accuracy: 73.50 -> 74.01 | Improved |
| add cluster-then-select on top of GVP on Qwen (ablation) | GVP · 7 general benchmarks (Qwen, 50 experts) | average accuracy: 55.56 -> 59.10 | Improved |
| use mixed-diversity calibration data instead of C4 for Enumeration Pruning (ablation) | Enumeration Pruning · 7 general benchmarks (Qwen, 50 experts) | average accuracy: 55.11 -> 49.47 | Hurt |
| use mixed calibration data instead of C4 for Enumeration on Mixtral (ablation) | Enumeration Pruning · 7 general benchmarks (Mixtral, 6 experts) | average accuracy: 72.73 -> 72.79 | No clear effect |
| use MoP instead of Enumeration Pruning at 4 experts | expert pruning · 7 general benchmarks (Mixtral, 4 experts) | average accuracy: 62.41 -> 68.96 | Improved |
| use MoP instead of Enumeration Pruning at 40 experts | expert pruning · 7 general benchmarks (Qwen, 40 experts) | average accuracy: 38.60 -> 55.74 | Improved |
| use MoP instead of Enumeration Pruning on code at 4 experts | expert pruning · HumanEval (Mixtral, 4 experts) | pass rate: 1.22 -> 71.34 | Improved |
| use MoP on MATH, Mixtral 4 experts | expert pruning · MATH (Mixtral, 4 experts) | accuracy: 3.80 -> 3.00 | Hurt |
| use MoP on MBPP, Qwen 50 experts | expert pruning · MBPP (Qwen) | pass rate: 8.17 -> 9.34 (40 experts: 8.17 vs 8.17 tie) | Mixed |
| remove 2 experts per layer with MoP (6 of 8) | MoP · Mixtral inference | memory GB / speedup: Mem 87.87 -> 66.87, 1.22x speedup | Improved |
Closest papers by meaning
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.91)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning (NeurIPS 2025 · similarity 0.90)
- Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025 · similarity 0.90)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
