AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks

Taishi Nakamura, Satoki Ishikawa, Masaki Kawamura, Takumi Okamoto, Daisuke Nohara, Jun Suzuki, Rio Yokota

ICLR 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase total parameters by adding experts (higher sparsity) at fixed width and top-k (ablation)Mixtral-style MoE (16 layers) · GSM8K and GSM-Plustask loss and accuracyHurt
increase total parameters by adding experts at fixed width and top-k (ablation)Mixtral-style MoE (16 layers) · TriviaQA and HellaSwagtask loss and accuracyImproved
increase top-k (active parameters) at matched pre-training loss or fixed TPP (ablation)Mixtral-style MoE · GSM8K and GSM-Plusreasoning accuracyImproved
change top-k while keeping active parameters constant (ablation)Mixtral-style MoE · memorization and reasoning benchmarksperformanceNo clear effect
make MoE denser at large active-parameter budgets (ablation)Mixtral-style MoE · GSM8K and GSM-Plus (fixed active params)task loss and accuracyImproved
increase sparsity at fixed active parameters (ablation)Mixtral-style MoE · TriviaQA and HellaSwagtask loss and accuracyImproved
tune tokens per parameter toward about 20 (ablation)Mixtral-style MoE · reasoning tasksaccuracy: peak near TPP of 20Improved
lower tokens per parameter (more parameters) (ablation)Mixtral-style MoE · TriviaQA and HellaSwagaccuracyImproved
add GRPO post-training on GSM8K (ablation)MoE models · GSM8KaccuracyNo clear effect
add test-time compute (self-consistency with 128 samples) (ablation)MoE models · GSM8K (zero-shot)accuracyNo clear effect
increase top-k only at inference time (ablation)MoE d=2048, 128 experts, top-2 · GSM8K with TTCPass@1No clear effect
lower learning rate and smaller initialization scale (ablation)MoE E=16, k=2 · reasoning benchmarks at matched training lossdownstream accuracyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy