Research map / Architectures and efficiency / Mixture of experts

How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives hyperparameter scaling (init variance, learning rates, Adam epsilon) for Mixture-of-Experts networks across width, expert width, expert count and sparsity regimes, extending muP to a "maximally scale-stable parameterization" (MSSP) and checking it with DMFT and coordinate checks.

Problem

Hyperparameters for MoE models (init, learning rates, Adam epsilon) lack a principled scaling rule with respect to network width, expert width, number of experts, sparsity and depth, so stable and optimal training at scale is not guaranteed.

Approach

Extends muP-style signal-propagation analysis to a minimal MoE block across asymptotic regimes where width, expert width and expert count diverge, deriving per-layer init variances and learning rates; proposes MSSP, a parameterization in which hidden-state and gradient scales are balanced across the expert and router pathways.

Key results

• In Regime II, muP-style init leaves expert-pathway and router-pathway gradients imbalanced (sqrt(N) deficit in W^{3,i} row), which MSSP removes by boosting W^{3,i} init variance to M/N_e and zeroing W^4 at init.
• Transformer MoE coordinate checks are reported as verifying that the block-level scaling composes within larger networks.

Surprising findings

• Under muP in Regime II, the W^{3,i} update pieces lag the propagating pieces by a sqrt(N) factor, so the router pathway is dominated by the initialization contribution.

Limitations

• Analysis uses leading-order coordinate-scale exponents and ignores Theta(1) prefactors.
• Derivation is for a minimal MoE block with sigmoid gating; softmax case is asserted rather than fully derived in the shown text.

Experimental setup

Metrics: coordinate scale exponents coordinate checks

Baselines: muP

Builds on: muP Tensor Programs

Closest papers by meaning

• Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · similarity 0.92)
• Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.92)
• Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025 · similarity 0.91)
• Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.90)
• Mixtures of Experts Unlock Parameter Scaling for Deep RL (ICML 2024 · similarity 0.90)
• Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)