Research map / Architectures and efficiency / Mixture of experts
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace a computationally matched LSTM baseline with a sparse MoE layer using 4096 experts | 1 Billion Word language model · 1 Billion Word test | perplexity: 45.0 -> 34.1 | Helped |
| increase MoE capacity to 65536 experts | language model · 100 Billion Word Google News, test (1 epoch) | perplexity: 47.0 -> 28.9 | Helped |
| increase capacity from 65536 to 131072 experts | language model · 100 Billion Word Google News, test (1 epoch) | perplexity: 28.9 -> 29.2 | Hurt |
| add a 2048-expert MoE to the translation model | GNMT-style En→Fr model · WMT'14 En→Fr newstest2014 | BLEU: +1.34 BLEU | Helped |
| add a 2048-expert MoE to the translation model | GNMT-style En→De model · WMT'14 En→De newstest2014 | BLEU: +1.12 BLEU | Helped |
| replace multilingual GNMT with the multilingual MoE model | multilingual translation · multilingual dev | perplexity: -19% | Helped |
| replace multilingual GNMT with the multilingual MoE model | multilingual translation · English→Korean test | BLEU: -1.79 | Hurt |
| increase computation from 8.4M to 142.7M ops/timestep for the high-capacity MoE | 1 Billion Word language model · 1 Billion Word test (10 epochs) | perplexity: 34.1 -> 28.0 | Helped |
| remove both importance and load balancing losses (ablation) | MoE-256 · 1 Billion Word test | perplexity: 35.6 -> 39.8 | Hurt |
| add importance balancing loss (ablation) | MoE-256 · 1 Billion Word test | perplexity: 39.8 -> 35.6 | Helped |
| add load balancing loss (ablation) | MoE-256 · 1 Billion Word test | perplexity: 39.8 -> 35.7 | Helped |
| increase the load-loss weight (ablation) | MoE-256 · 1 Billion Word expert utilization | maximum-to-mean expert load: 1.15 -> 1.07 | Helped |
Closest papers by meaning
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.90)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.90)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.90)
- Mixture of Experts Made Intrinsically Interpretable (ICML 2025 · similarity 0.90)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.