Research map / Architectures and efficiency / Mixture of experts
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically studies how MoE sparsity trades off total parameters against active FLOPs per token, fitting isoFLOP surfaces and a sparsity-aware scaling law to find optimal sparsity under fixed parameter and compute budgets.
Problem
Scaling laws for language models typically equate capacity with parameter count, but in sparse MoEs parameters and FLOPs per example decouple; the optimal trade-off between total parameters and active compute under a fixed training budget is unclear.
Approach
Define MoE sparsity as the fraction of inactive experts, fit a single 3D isoFLOP surface over model size and sparsity (quadratic polynomial in log space), and extend the Chinchilla-style loss law with sparsity terms to predict loss and optimal sparsity for fixed compute and parameter budgets.</201>
Key results
- Under fixed training compute, sparser MoEs reach lower pretraining loss, with optimal total parameters increasing and optimal active parameters decreasing as sparsity increases.
- For a fixed model size, optimal sparsity increases with model size and converges toward 1.0 for large models; smaller models (around 500M parameters or less) stay dense for most budgets.
- Downstream few-shot error tracks upstream loss for most tasks regardless of sparsity, but reading comprehension tasks (e.g., SQuAD, CoQA) favor denser models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase MoE sparsity while allowing total parameter count to grow | Compute-optimal MoE language models · RedPajamaV1 pretraining | pretraining loss | Helped |
| Increase sparsity at fixed total parameter count | MoE language models · Fixed training compute budget | pretraining loss | Mixed |
| Increase total parameter count | Compute-optimal MoE language models · Fixed training compute budget | optimal sparsity | Helped |
| Increase training compute budget at fixed model size | MoE language models · Fixed total parameter count | optimal sparsity | Hurt |
| Increase sparsity in models matched on pretraining loss | MoE language models · Reading comprehension | downstream performance | Hurt |
| Vary sparsity at similar pretraining performance | MoE language models · Language understanding and world knowledge few-shot tasks | downstream performance | No clear effect |
| Add chain-of-thought prompting to increase inference-time compute | MoE language models compared with dense models of similar active parameter count · Reasoning tasks | downstream performance | Helped |
| Add router jitter noise (ablation) | MoE language models · Pretraining | pretraining loss | No clear effect |
Surprising findings
- On reading comprehension tasks, sparser MoEs transfer worse than denser models at matched pretraining loss, possibly due to lower inference compute per example.
- MoEs benefit more from chain-of-thought prompting than dense models with similar active parameter counts.
Limitations
- Costs measured only in theoretical FLOPs, ignoring memory and communication overhead.
- Experiments are compute-optimal small-scale models; results may be artifacts of scale.
- Reading comprehension transfer gap explanation is not verified.
Experimental setup
Datasets: RedPajamaV1
Benchmarks: PIQA CommonSenseQA OpenBookQA COPA LAMBADA HellaSwag Winograd Winogrande SQuAD CoQA BoolQ TruthfulQA
Metrics: pretraining loss (cross-entropy) downstream few-shot error
Baselines: dense transformers
Builds on: Chinchilla scaling laws Megablocks scaling laws for MoE (granularity) isoFLOP approach
Closest papers by meaning
- Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · similarity 0.92)
- How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization (NeurIPS 2026 · similarity 0.92)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.91)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.91)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.91)
- Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.