Research map / Architectures and efficiency / Mixture of experts
Mixture-of-Experts with Expert Choice Routing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| route tokens by having experts select their top-scoring tokens, allowing a variable number of experts per token | MoE models compared with GShard Top-2 · GLUE/SuperGLUE selected tasks, dev (8B/64E) | Avg acc.: 90.3 -> 92.6 | Helped |
| use expert-choice routing instead of dense feed-forward layers | 8B model · GLUE/SuperGLUE selected tasks, dev | Avg acc.: 89.2 -> 92.6 | Helped |
| use expert-choice routing instead of Hash Layer routing | MoE fine-tuning · Selected GLUE/SuperGLUE tasks | Avg acc.: 81.3 ± 0.1 -> 84.0 ± 0.2 | Helped |
| cap each token at 2 experts instead of allowing a variable number (ablation) | Expert Choice · Selected GLUE/SuperGLUE tasks | Avg acc.: 84.0 ± 0.2 -> 83.2 ± 0.4 | Hurt |
| cap each token at 3 experts instead of allowing a variable number (ablation) | Expert Choice · Selected GLUE/SuperGLUE tasks | Avg acc.: 84.0 ± 0.4 vs. 84.0 ± 0.2 | No clear effect |
| reduce the capacity factor from 2 to 1 (ablation) | Expert Choice · Pre-training evaluation | perplexity | Hurt |
| reduce the capacity factor from 2 to 0.5 (ablation) | Expert Choice · Pre-training evaluation | perplexity | Mixed |
| increase the number of experts while keeping expert size fixed (ablation) | Expert Choice and GShard Top-2 · Pre-training evaluation dataset | perplexity | Helped |
Closest papers by meaning
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.92)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.90)
- Autonomy-of-Experts Models (ICML 2025 · similarity 0.89)
- ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing (ICLR 2025 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Mixture of Lookup Experts (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.