Research map / Architectures and efficiency / Mixture of experts
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives joint scaling laws for dense and Mixture-of-Experts language models over active parameters, tokens and expert count, and uses them to show MoE can be more memory-efficient than dense models under fixed memory and compute budgets.
Problem
Scalability and efficiency of MoE models under memory constraints (total parameters, KV cache, inference cost) are underexplored, and it is unclear how many experts are optimal for a given compute and memory budget.
Approach
A joint scaling law L(N_act, D, E_hat) that extends Chinchilla-style fits with expert-count power-law terms and interaction exponents in ln(E), fitted across over 280 dense and MoE runs, then used to derive compute-, memory-, and inference-optimal expert counts, plus a learning-rate scaling rule that depends on expert count.
Key results
- The fitted joint scaling law achieves RMSE of 0.0039 on held-out runs and 0.0062 on training runs.
- Under a 1.1B total-parameter budget, compute-matched 2- and 4-expert MoE models attain lower loss than an overtrained 1.1B dense model.
- Rule of thumb: for fixed total parameters, an MoE with E<=8 experts beats a compute-optimal dense model when trained on E times more tokens at the same memory footprint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a memory- and compute-matched MoE instead of a dense model | 1.1B-parameter models · Same compute and total memory budgets | Final loss | Helped |
| Use a memory- and compute-matched MoE instead of a dense model | 1.1B-parameter models · Inference | Performance | Helped |
| Increase the number of experts in the compute-optimal configuration | MoE scaling law · Fixed training compute budget | Compute-optimal active parameters | Hurt |
| Increase the number of experts in the compute-optimal configuration | MoE scaling law · Fixed training compute budget | Compute-optimal training tokens | Helped |
| Use E=2 instead of a dense model | MoE inference · Inference | FLOPs per token: 36% | Helped |
| Use E=4 instead of a dense model | MoE inference · Inference | FLOPs per token: 61% | Helped |
| Train an MoE with E≤8 on E times more tokens while maintaining the same memory footprint | Compute-optimal dense model · Fixed total parameter count | Final loss | Helped |
| Include the number of experts in the learning-rate scaling rule (ablation) | MoE learning-rate scaling · Extrapolation to E=32 | Learning-rate prediction performance | Helped |
| Omit the number of experts from the learning-rate scaling rule (ablation) | MoE learning-rate scaling · Extrapolation to E=32 | Learning-rate prediction performance | Hurt |
| Use the joint scaling law instead of separate Chinchilla fits by expert count | Scaling-law fit · Training dataset | RMSE: 0.0059 -> 0.0062 | Hurt |
| Use the joint scaling law instead of separate Chinchilla fits by expert count | Scaling-law fit · Held-out validation set | RMSE: 0.0041 -> 0.0039 | Helped |
Surprising findings
- MoE models can be more memory-efficient than dense models with the same total parameters and training loss, contrary to common belief that MoE is memory-inefficient.
- The optimal token-to-parameter ratio and dataset exponent change substantially with the number of experts.
Limitations
- Uses standard non-granular MoE with expert size equal to dense FFN; fine-grained MoE and dropless variants not studied.
- FLOPs-based analysis may not reflect wall-clock training time; implementation and distributed training differences are not considered.
- Relies on Chinchilla functional form, which is less reliable at extreme token-to-parameter ratios; relies on large dataset availability.
Experimental setup
Datasets: FineWeb-Edu
Metrics: final training loss RMSE of fitted scaling law Huber loss
Baselines: compute-optimal dense models Clark et al. MoE scaling law
Builds on: Chinchilla scaling laws Switch Transformer Clark et al. MoE scaling laws SwiGLU
Closest papers by meaning
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.91)
- Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · similarity 0.91)
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.91)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.91)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
- How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.