Research map / Architectures and efficiency / Mixture of experts
Scaling Laws for Fine-Grained Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces granularity as a MoE hyperparameter and fits joint scaling laws in N, D and G, showing fine-grained MoE beats dense Transformers at every compute budget under compute-optimal training.
Problem
Existing MoE scaling laws assume fixed training duration and fixed expert size (equal to the feed-forward layer), leading to claims that MoE loses to dense models at scale.
Approach
Introduce granularity G = d_ff / d_expert to control expert size independently of expansion rate, and derive a joint scaling law L(N, D, G) with variable training tokens, then solve for compute-optimal N, D, G under a FLOPs model that includes routing overhead.
Key results
- Compute-optimal MoE at 1e20 FLOPs matches a dense Transformer trained with ~20x more compute, with savings exceeding 40x beyond 1e25 FLOPs.
- Fitted joint scaling law for MoE with RMSE 0.015 (validation RMSE 0.019); optimal granularity increases with compute budget (G=8 at 2.95e18 FLOPs to G=64 at 5.69e24 FLOPs).
- Setting expert size equal to the feed-forward layer (G=1) is almost never compute-optimal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase granularity above G=1 (ablation) | MoE · C4 language modeling | loss | Helped |
| Use G=8 instead of G=16 (ablation) | MoE with N=64 x 7M, D=66B · Training loss versus wall-clock time on NVIDIA A100 | loss | Helped |
| Increase granularity to G=64 at d_model=256 and E=64 (ablation) | MoE · C4 language modeling | performance | Hurt |
| Use compute-optimal MoE instead of dense Transformer | Language modeling at a compute budget of 10^20 FLOPs · Compute-optimal scaling comparison | compute required for same quality: 20× | Helped |
| Use compute-optimal MoE instead of dense Transformer | Language modeling above a compute budget of 10^25 FLOPs · Compute-optimal scaling comparison | compute savings: 40× | Helped |
| Vary training duration and select compute-optimal N, D, and G | MoE versus dense Transformers · Compute-optimal scaling across budgets | efficiency | Helped |
| Increase the compute budget and use larger optimal granularity (ablation) | MoE, E=16 · Compute-optimal configurations | optimal granularity | Helped |
| Exclude the top 20% of models with lowest perplexity from scaling-law fitting (ablation) | MoE scaling law · Scaling-law validation | RMSE: 0.015 -> 0.019 | Mixed |
Surprising findings
- Contrary to prior work, MoE can be considered more efficient than dense Transformers at every model size when training hyperparameters are compute-optimal.
- Extremely high granularity (G=64 at d_model=256) shows performance decline when routing parameters exceed expert active parameters.
Limitations
- Expansion rate fixed at E=64 in main experiments; E=16 only in appendix with wider uncertainty.
- Cost modeled in FLOPs, ignoring communication and memory overheads of high granularity.
- Experiments limited to C4 with models up to 3.7B parameters, extrapolated to 1T.
Experimental setup
Datasets: C4
Metrics: validation loss (perplexity-based) RMSE of scaling law fit wall-clock training loss bootstrapped 10th/90th percentiles of optimal configs
Baselines: dense Transformer vanilla MoE (G=1) Clark et al. MoE scaling law
Builds on: Chinchilla scaling laws Expert Choice routing Clark et al. MoE scaling laws
Closest papers by meaning
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.92)
- How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization (NeurIPS 2026 · similarity 0.92)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.92)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.92)
- Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025 · similarity 0.92)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.