Research map / Architectures and efficiency / Mixture of experts
MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
Geng Zhang, Han Yuxuan, Yuxuan Lou, Yiqi Zhang, Wangbo Zhao, Yang You
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace pruned experts with constant novices and score by variance x frequency (MoNE) vs baselines | MoNE · 9-task zero-shot avg, 25% pruning, OLMoE | avg accuracy: 58.32 (FLAP) -> 61.04 | Improved |
| MoNE vs best baseline at 25% pruning on Qwen2-57B-A14B | MoNE · 9-task zero-shot avg, 25% pruning, Qwen2-57B-A14B | avg accuracy: 71.78 (FLAP) vs 71.75 | No clear effect |
| MoNE vs RS at 50% pruning on Qwen3-30B-A3B | MoNE · 9-task zero-shot avg, 50% pruning, Qwen3-30B-A3B | avg accuracy: 68.06 (RS) vs 66.32 | Hurt |
| use MoNE at 50% pruning instead of FLAP/RS/MC-SMoE | MoNE · 9-task zero-shot avg, 50% pruning, DeepSeek-V2-Lite | avg accuracy: 56.79 (FLAP) -> 60.40 | Improved |
| larger MoE models tolerate more pruning with MoNE | MoNE · max pruning ratio within 1% loss | max pruning ratio: 16% (OLMoE) -> 25% (Qwen2-57B-A14B) | Improved |
| continued pretraining on 2B tokens after pruning | MoNE on OLMoE 25% pruned · 9-task zero-shot avg, OLMoE | avg accuracy: 61.04 -> 64.18 | Improved |
| calibrate on task-specific data instead of Zyda2 pretraining data | MoNE on OLMoE-Instruct · GSM8K | accuracy: 7.96 -> 67.48 | Improved |
| calibrate on Zyda2 instead of no pruning for Math/GSM8K | MoNE · GSM8K, OLMoE base | accuracy: 52.77 -> 3.34 | Hurt |
| replace product score with weighted-sum score (ablation) | MoNE redundancy score · 9-task avg, OLMoE | avg accuracy: 61.04 -> 57.07 | Hurt |
| replace product score with weighted-sum score (ablation) | MoNE redundancy score · 9-task avg, Qwen2-57B-A14B | avg accuracy: 71.75 -> 72.55 | Improved |
| use log-sum instead of product score (ablation) | MoNE redundancy score · 9-task avg, OLMoE | avg accuracy: 61.04 -> 61.23 | No clear effect |
| fused redundancy score plus novice replacement vs ablated variants (ablation) | MoNE · 9 tasks, averaged over dimensions | avg accuracy: gains up to 8.85 on BoolQ/COPA/PIQA; frequency-only slightly better on MMLU | Mixed |
Later papers that compared with this method
2 later accepted papers report MoNE as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.
- Less Token, More Signal: MoE Expert Pruning via Critical Token Selection (ICML 2026 · better on 9 of 9 shared results)
- Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression (ICML 2026 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.93)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.92)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.91)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.91)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
