AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts

Ruizhe Wang, Yucheng Ding, Xiao Liu, Yaoxiang Wang, Peng CHENG, Baining Guo, Zheng-Jun Zha, Yeyun Gong

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use interposition instead of stacking for layer growth (ablation)depth growth of converged MoE · 3B to 6B MoE, convergedtraining loss and accuracyImproved
use interposition instead of stacking (ablation)depth growth · 17B MoE, 13x Chinchilla FLOPstraining loss and accuracyImproved
use interposition instead of stacking with growth factor 4 (ablation)depth growth · 3B MoE expanded 20 to 80 layerstraining loss and accuracyImproved
use interposition instead of stacking before the Chinchilla-optimal FLOPs (ablation)depth growth · early-stage checkpointsaccuracyNo clear effect
add small Gaussian noise (alpha=0.01) to copied experts (ablation)width growth · 3B MoE width growthdownstream accuracy: about +1% accuracyImproved
add excessive noise to copied experts (ablation)width growth · 3B MoE width growthdownstream accuracyHurt
add noise to copied experts (ablation)width growth · 3B MoE width growthlanguage modeling lossNo clear effect
randomly initialize new experts instead of noisy copies (ablation)width growth · 3B MoE width growthdownstream accuracyHurt
use depth growth instead of width growthorthogonal growth · 3B MoE single-axis growthdownstream accuracyImproved
change order of depth and width growthorthogonal growth · 3B MoE dual growthdownstream accuracyNo clear effect
grow from a later checkpoint with more sunk cost (8k to 88k steps), fixed extra FLOPsdepth growth · 3B MoEaverage downstream accuracy: 39.09 -> 47.88 (scratch 36.29)Improved
grow from 96k (annealing) instead of 88k stepsdepth growth · 3B MoEaverage downstream accuracy: 47.88 -> 47.82No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy