AI Paper Atlas

Research map / Learning theory and algorithms / Kernels and regression theory

Generalization and Scaling Laws for Mixture-of-ExpertsTransformers

Mansour MAYAKI

ICML 2026 · Learning theory and algorithms › Kernels and regression theory · approach: Generalization theory of deep networks

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase expert pool M from 16 to 64 at fixed k=4 (ablation)MoE LM · routing ablationvalidation loss: 2.73 -> 2.37Improved
increase routing term k log(eM/k) in moderate regime (M/k <= 8) (ablation)MoE LM · routing ablationvalidation lossHurt
increase number of experts E from 1 to 32 (ablation)MoE scaling laws (JMSL data) · per-E identity checkresidual r_D: 0.0427 -> 0.1304Hurt
increase number of experts E from 1 to 32 (ablation)MoE scaling laws (JMSL data) · per-E identity checkratio alpha_D/alpha_N: 1.08 -> 1.70Improved
increase E from 1 to 32 in amplification kappa (ablation)MoE scaling laws (JMSL data) · per-E identity checkkappa(E): 1.28 -> 1.97Improved
raise beta from 1.0 to 1.5 in theoryexponent fit · OpenWebTextalpha_N: 0.044 -> 0.067Improved
measure model exponent against active parametersMoE LM · WikiText-103alpha_N empirical vs theory: 0.063 -> 0.060No clear effect
measure data exponent against theoryMoE LM · TinyStoriesalpha_D empirical vs theory: 0.080 -> 0.081No clear effect
raise compute-optimal tokens per active parameter via more experts, E from 1 to 16 at budget 1e20 vs 1e21 (ablation)JMSL allocations · compute-optimal D*/N*D*/N* gain: x4.5 vs x3.0Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy