Research map / Architectures and efficiency / Mixture of experts
Multilingual Routing in Mixture-of-Experts
Lucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu, Nanyun Peng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| steer router toward English-active math experts in middle layers (soft, lambda 0.5) | Qwen3-30B-A3B · MGSM non-English avg | accuracy: 76.4% -> 78.0% | Improved |
| steer router toward English-active math experts in middle layers (soft, lambda 0.5) | Phi-3.5-MoE · MGSM non-English avg | accuracy: 57.5% -> 58.9% | Improved |
| force-activate English-active math experts in middle layers (hard) | GPT-OSS-20B · MGSM non-English avg | accuracy: 68.9% -> 71.5% | Improved |
| force-activate English-active medicine experts in middle layers (hard) | Qwen3-30B-A3B · Global-MMLU Medicine non-English avg | accuracy: 68.2% -> 69.1% | Improved |
| steer toward English-active medicine experts (soft, lambda 0.5) | Phi-3.5-MoE · Global-MMLU Medicine non-English avg | accuracy: 57.8% -> 58.8% | Improved |
| steer toward English-active medicine experts (soft, lambda 0.5) | GPT-OSS-20B · Global-MMLU Medicine non-English avg | accuracy: 63.8% -> 64.5% | Improved |
| apply steering to low-resource languages only average | GPT-OSS-20B · MGSM low-resource avg | accuracy: 60.0% -> 64.5% | Improved |
| apply steering to English input | Phi-3.5-MoE · MGSM English | accuracy: 88.0% -> 85.6% | Hurt |
| deactivate multilingual or task experts in all layers or random subsets (ablation) | MoE LLMs · MGSM and Global-MMLU | accuracy | Hurt |
| intervene in layers slightly outside the identified middle zone (ablation) | MoE LLMs · MGSM and Global-MMLU | accuracy | Hurt |
| combine task-expert steering with multilingual-expert activation in top and bottom layers (ablation) | MoE LLMs · MGSM and Global-MMLU | accuracy | Hurt |
| use larger soft steering strength than 0.5 (ablation) | MoE LLMs · MGSM and Global-MMLU | accuracy | Hurt |
Closest papers by meaning
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.93)
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.92)
- OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.91)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.90)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
- Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
