Research map / Architectures and efficiency / Mixture of experts
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
William Fedus, Barret Zoph, Noam Shazeer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| route to a single expert (Switch, k=1) instead of top-2 MoE at capacity factor 1.0 | Switch-Base · C4 pre-training | time to quality threshold (hours): 80.1 -> 62.8 | Improved |
| lower the capacity factor from 2.0 to 1.0 (ablation) | Switch-Base · C4 pre-training | time to quality threshold (hours): 72.8 -> 62.8 | Improved |
| selectively cast router to float32 within bfloat16 training (ablation) | Switch-Base · C4 pre-training (32 experts) | neg. log perplexity: -3.780 -> -1.716 | Improved |
| reduce initialization scale by 10x (0.1x-init) (ablation) | Switch-Base · C4 pre-training (32 experts, 3.5k steps) | neg. log perplexity: -3.60 -> -2.72 | Improved |
| add expert dropout of 0.4 on expert layers with 0.1 elsewhere (ablation) | Switch-Base · GLUE | average score: 84.7 -> 85.2 | Improved |
| raise dropout to 0.3 at all layers (ablation) | Switch-Base · GLUE | average score: 84.7 -> 83.9 | Hurt |
| initialize dense student with non-expert weights of the teacher (ablation) | Distilled T5-Base · C4 distillation | neg. log perplexity: -1.631 -> -1.598 | Improved |
| mix 0.25 teacher probabilities with 0.75 ground truth label in distillation loss (ablation) | Distilled T5-Base · C4 distillation | neg. log perplexity: -1.598 -> -1.580 | Improved |
| scale to 64 experts at fixed FLOPs per token (ablation) | Switch-Base · C4 pre-training | step speedup over T5-Base: ~7.5x speedup | Improved |
| sample expert from the router softmax instead of argmax (ablation) | Switch Transformer · C4 pre-training | neg. log perplexity: -1.471 -> -1.570 | Hurt |
| add multiplicative input jitter noise to the router (ablation) | Switch Transformer · C4 pre-training | neg. log perplexity: -1.471 -> -1.468 | Improved |
| reroute overflow tokens to second-highest expert (No-Token-Left-Behind) (ablation) | Switch Transformer · C4 pre-training | quality / stability | No clear effect |
Closest papers by meaning
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.90)
- Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.89)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.88)
- Mixture of Tokens: Continuous MoE through Cross-Example Aggregation (NeurIPS 2024 · similarity 0.88)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
