Research map / Large language models / Language, safety and interpretability
Scaling Laws for Upcycling Mixture-of-Experts Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies scaling laws for upcycling dense LLMs into Mixtral-style MoE models, fitting a joint law in dense-pretraining tokens, upcycled tokens and model size, and derives when upcycling beats from-scratch MoE training.
Problem
Upcycling dense checkpoints into MoE models is used for efficient LLM pretraining, but its effectiveness and limits are unclear, with conflicting reports and no guidance on when to upcycle.
Approach
Proposes a multiplicative power-law with an interaction term in the exponent, L = A D1^{-a1} D2^{-a2 + a3 log D1} + E, capturing how the dense pretraining budget (sunk cost) reduces initial loss but slows subsequent gains from upcycled tokens; uses WSD schedule checkpoints to emulate many token budgets with one sweep.
Key results
- Upcycling exponent interaction term fitted; multiplicative form with interaction achieves lowest leave-one-out RMS error across four MoE configurations (e.g., 0.0031 for (8,2)).
- Joint upcycling scaling law extrapolates with validation RMS error of 0.015; fitted irreducible loss E = 0.165.
- Upcycling is more efficient than from-scratch MoE training only when D2 is below a threshold D* (4B tokens for 8x1B); compute-optimal upcycling implies D2 scales as N1^1.8 for Llama2-sized models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| upcycle a 1B dense model into an 8x1B MoE | 1B dense model · ARC-c, ARC-e, sciq, winogrande, and average benchmark accuracy | accuracy: ARC-c: 27.65 -> 30.12; ARC-e: 52.10 -> 56.14; sciq: 78.10 -> 82.00; winogrande: 54.93 -> 57.77; Avg.: 49.98 -> 52.94 | Helped |
| upcycle a 1B dense model into an 8x1B MoE | 1B dense model · lambada, logiqa, and piqa | accuracy: lambada: 53.52 -> 49.72; logiqa: 29.49 -> 27.65; piqa: 69.31 -> 67.19 | Hurt |
| use WSD instead of cosine learning-rate schedule (ablation) | dense and MoE training · language-model validation loss | cross-entropy loss | No clear effect |
| use the MoE learning rate for upcycled training rather than the dense-training or final dense constant learning rate (ablation) | upcycled 8x0.1B MoE · Slimpajama validation | cross-entropy loss | Helped |
| add Gaussian noise to the copied expert weights (ablation) | upcycled MoE · upcycled training | cross-entropy loss | No clear effect |
| partially randomize the copied MLP weights (ablation) | upcycled MoE · upcycled training | cross-entropy loss | No clear effect |
| replace MLP weights with randomly selected low-rank counterparts (ablation) | upcycled MoE · upcycled training | cross-entropy loss | No clear effect |
| set the auxiliary load-balancing coefficient to 1 rather than around 10^-3 (ablation) | 8x0.1B MoE · Slimpajama validation | cross-entropy loss | Hurt |
| use distinct, non-overlapping datasets for dense and upcycled training instead of reusing the same dataset (ablation) | 8x0.1B MoE · Slimpajama validation | cross-entropy loss | No clear effect |
| include the interaction term α3 log D1 in the multiplicative dataset scaling law (ablation) | upcycled MoE dataset scaling law · 0.1B dense model upcycled to MoE; architectures (4,1), (4,2), (8,1), (8,2) | leave-one-out RMS error: (4,1): 0.0169 -> 0.0111; (4,2): 0.0085 -> 0.0081; (8,1): 0.0180 -> 0.0105; (8,2): 0.0095 -> 0.0031 | Helped |
| include the interaction term in the model-configuration scaling fit (ablation) | upcycled MoE model-configuration scaling law · model configuration scaling data | leave-one-out RMS error: multiplicative: 0.0351 -> 0.0414; additive: 0.0322 -> 0.0341 | Hurt |
| increase the auxiliary load-balancing loss coefficient from around 10^-3 (ablation) | 8x0.1B MoE · expert load balancing and validation cross-entropy | cross-entropy loss | Mixed |
Surprising findings
- Upcycling is less compute-efficient than training compute-optimal dense or MoE models from scratch under a fixed FLOPs budget.
- Alternative upcycling tweaks (noise injection, partial randomization, low-rank substitution) gave no advantage over direct weight copying.
Limitations
- Interaction term lacks theoretical justification
- Model configuration scaling limited by compute; no unified scaling law across dense size, sparsity and token budgets
- Experiments limited to models up to 1B dense and 7B-equivalent MoE scale
Experimental setup
Datasets: SlimPajama-DC CommonCrawl Japanese CommonCrawl Stack code dataset
Benchmarks: ARC-c ARC-e lambada logiqa piqa sciq winogrande
Metrics: cross-entropy test loss RMS leave-one-out fit error accuracy (normalized)
Baselines: from-scratch dense training from-scratch MoE training Pythia-1B TinyLlama-1.1B
Builds on: Chinchilla scaling law sparse upcycling Mixtral Megatron-LM
Closest papers by meaning
- Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · similarity 0.92)
- How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization (NeurIPS 2026 · similarity 0.91)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.91)
- Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization (ICLR 2025 · similarity 0.91)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.