Research map / Architectures and efficiency / Mixture of experts
Breaking the Echo Chamber: A Dynamic Ensemble Pruning Perspective on MoE
Xinlai Kang, Dunyao Xue, Zhengbo Wang, Chengshuo Du, Xinghao Chen, Hang Zhou, Hanting Chen, Cheng Meng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Mahalanobis diversity objective instead of l1-norm criterion (Sigma = identity) (ablation) | MP-MoE on OLMoE-1B-7B · OLMoE-1B-7B ablation | benchmark performance: -3.00% | Improved |
| use normalized covariance estimator instead of raw co-occurrence matrix C (ablation) | MP-MoE on OLMoE-1B-7B · OLMoE-1B-7B ablation | benchmark performance: -1.37% | Improved |
| add MP warmup schedule (standard routing for first 1% of steps) (ablation) | MP-MoE on OLMoE-1B-7B · OLMoE-1B-7B ablation | benchmark performance: -0.55% | Improved |
| replace top-k routing with MP-MoE routing | OLMoE-1B-7B, 1B tokens · Six-benchmark average | average score: 0.282 -> 0.312 | Improved |
| replace top-k routing with MP-MoE routing | OLMoE-1B-7B, 5B tokens · Six-benchmark average | average score: 0.294 -> 0.319 | Improved |
| replace top-k routing with MP-MoE routing | OLMoE-0.2B-1B, 50B tokens · Six-benchmark average | average score: 0.279 -> 0.291 | Improved |
| replace top-k routing with MP-MoE routing | OLMoE-0.2B-1B, 50B tokens · HellaSwag | accuracy: 0.246 -> 0.243 | No clear effect |
| replace top-k routing with MP-MoE routing | OLMoE-0.2B-1B, 50B tokens · ARC-Challenge | accuracy: 0.255 -> 0.257 | No clear effect |
| replace top-k routing with MP-MoE routing | OLMoE-1B-7B, 1B tokens · BBH | exact match: 0.121 -> 0.169 | Improved |
| replace top-k routing with MP-MoE routing | OLMoE-1B-7B, 1B tokens · BoolQ | accuracy: 0.563 -> 0.597 | Improved |
| replace top-k routing with MP-MoE routing | Flame-MoE (38M active parameters) · pre-training perplexity | PPL: 4.71% reduction in PPL | Improved |
| replace top-k routing with SimSMoE (competing diversity method) | Flame-MoE (38M active parameters) · pre-training perplexity | PPL: 1.31% reduction | Improved |
Closest papers by meaning
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.91)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.91)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.91)
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning (NeurIPS 2025 · similarity 0.90)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
