Research map / Architectures and efficiency / Mixture of experts
Understanding Routing Mechanism in Mixture-of-Experts Language Models
Wengang Li, Lingqi Zhang, Toshio Endo, Mohamed Wahib
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| zero the routing scores assigned by expert M1E9 (ablation) | OLMoE · OLMoE routing score variance | variance stripe at sending layer M1 | Hurt |
| zero the routing scores assigned by expert M4E14 (ablation) | OLMoE · OLMoE routing score variance | variance stripe at sending layer M4 | Hurt |
| zero the scores assigned by M1E18 (control expert) (ablation) | OLMoE · OLMoE routing score variance | variance stripes | No clear effect |
| pad output of M1E9 with zeros (ablation) | OLMoE · OLMoE routing score variance | variance stripes | Hurt |
| pad output of M4E14 with zeros (ablation) | OLMoE · OLMoE routing score variance | variance stripes | Mixed |
| pad output of M1E18 with zeros (ablation) | OLMoE · OLMoE routing score variance | variance stripes | No clear effect |
Closest papers by meaning
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.91)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.90)
- OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.90)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.89)
- Ada-K Routing: Boosting the Efficiency of MoE-based LLMs (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
