Research map / Architectures and efficiency / Mixture of experts
Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
Ruizhe Wang, Yucheng Ding, Xiao Liu, Yaoxiang Wang, Peng CHENG, Baining Guo, Zheng-Jun Zha, Yeyun Gong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use interposition instead of stacking for layer growth (ablation) | depth growth of converged MoE · 3B to 6B MoE, converged | training loss and accuracy | Improved |
| use interposition instead of stacking (ablation) | depth growth · 17B MoE, 13x Chinchilla FLOPs | training loss and accuracy | Improved |
| use interposition instead of stacking with growth factor 4 (ablation) | depth growth · 3B MoE expanded 20 to 80 layers | training loss and accuracy | Improved |
| use interposition instead of stacking before the Chinchilla-optimal FLOPs (ablation) | depth growth · early-stage checkpoints | accuracy | No clear effect |
| add small Gaussian noise (alpha=0.01) to copied experts (ablation) | width growth · 3B MoE width growth | downstream accuracy: about +1% accuracy | Improved |
| add excessive noise to copied experts (ablation) | width growth · 3B MoE width growth | downstream accuracy | Hurt |
| add noise to copied experts (ablation) | width growth · 3B MoE width growth | language modeling loss | No clear effect |
| randomly initialize new experts instead of noisy copies (ablation) | width growth · 3B MoE width growth | downstream accuracy | Hurt |
| use depth growth instead of width growth | orthogonal growth · 3B MoE single-axis growth | downstream accuracy | Improved |
| change order of depth and width growth | orthogonal growth · 3B MoE dual growth | downstream accuracy | No clear effect |
| grow from a later checkpoint with more sunk cost (8k to 88k steps), fixed extra FLOPs | depth growth · 3B MoE | average downstream accuracy: 39.09 -> 47.88 (scratch 36.29) | Improved |
| grow from 96k (annealing) instead of 88k steps | depth growth · 3B MoE | average downstream accuracy: 47.88 -> 47.82 | No clear effect |
Closest papers by meaning
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
- Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025 · similarity 0.90)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.90)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.90)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.90)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
