AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

William Fedus, Barret Zoph, Noam Shazeer

JMLR 2022 · Architectures and efficiency › Mixture of experts · approach: Sparse MoE as regularization and attention · 1,668 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
route to a single expert (Switch, k=1) instead of top-2 MoE at capacity factor 1.0Switch-Base · C4 pre-trainingtime to quality threshold (hours): 80.1 -> 62.8Improved
lower the capacity factor from 2.0 to 1.0 (ablation)Switch-Base · C4 pre-trainingtime to quality threshold (hours): 72.8 -> 62.8Improved
selectively cast router to float32 within bfloat16 training (ablation)Switch-Base · C4 pre-training (32 experts)neg. log perplexity: -3.780 -> -1.716Improved
reduce initialization scale by 10x (0.1x-init) (ablation)Switch-Base · C4 pre-training (32 experts, 3.5k steps)neg. log perplexity: -3.60 -> -2.72Improved
add expert dropout of 0.4 on expert layers with 0.1 elsewhere (ablation)Switch-Base · GLUEaverage score: 84.7 -> 85.2Improved
raise dropout to 0.3 at all layers (ablation)Switch-Base · GLUEaverage score: 84.7 -> 83.9Hurt
initialize dense student with non-expert weights of the teacher (ablation)Distilled T5-Base · C4 distillationneg. log perplexity: -1.631 -> -1.598Improved
mix 0.25 teacher probabilities with 0.75 ground truth label in distillation loss (ablation)Distilled T5-Base · C4 distillationneg. log perplexity: -1.598 -> -1.580Improved
scale to 64 experts at fixed FLOPs per token (ablation)Switch-Base · C4 pre-trainingstep speedup over T5-Base: ~7.5x speedupImproved
sample expert from the router softmax instead of argmax (ablation)Switch Transformer · C4 pre-trainingneg. log perplexity: -1.471 -> -1.570Hurt
add multiplicative input jitter noise to the router (ablation)Switch Transformer · C4 pre-trainingneg. log perplexity: -1.471 -> -1.468Improved
reroute overflow tokens to second-highest expert (No-Token-Left-Behind) (ablation)Switch Transformer · C4 pre-trainingquality / stabilityNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy