AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

SNaX: sparse narrow accelerated mixture of experts

Wentao Guo, Mayank Mishra, Xinle Cheng, Ion Stoica, Tri Dao

ICLR 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add token rounding routing instead of vanilla top-K token choiceMoE training (TR) · 1.8B MoE, 40B tokens, 8/256 activated, avg accuracyAvg accuracy: 52.8 -> 53.5Improved
add token rounding instead of top-K token choiceMoE training (TR) · 1.4B MoE, 100B tokens, 2/128 activated, validation perplexityvalidation perplexity: 13.32 -> 13.22Improved
add token rounding instead of top-K token choiceMoE training (TR) · 1.4B MoE, 100B tokens, 2/128 activated, avg accuracyAvg accuracy: 52.0 -> 52.0No clear effect
use expert-choice routing instead of token rounding, then evaluate with TCMoE training · 0.5B MoE, 40B tokens, validation perplexityvalidation perplexity: 15.92 -> 18.61Hurt
fine-tune a TC router instead of using an auxiliary router for ECEC routing · 0.5B MoE, 40B tokens, validation perplexityvalidation perplexity: 21.80 -> 16.98Improved
add token rounding instead of token droppingMoE training · 1.4B MoE, 100B tokens, avg accuracyAvg accuracy: 51.3 -> 52.0Improved
always round expert frequency down (DOWN) instead of nearest rounding (ablation)token rounding · 0.5B MoE, 40B tokens, validation perplexityvalidation perplexity: 15.92 -> 16.10Hurt
always round expert frequency up instead of nearest rounding (ablation)token rounding · 0.5B MoE, 40B tokens, validation perplexityvalidation perplexity: 15.92 -> 15.89No clear effect
use stochastic rounding instead of nearest rounding (ablation)token rounding · 0.5B MoE, 40B tokens, validation perplexityvalidation perplexity: 15.92 -> 15.93No clear effect
increase expert granularity from n=1024 to n=256 at constant FLOPs (ablation)MoE · 0.5B MoE, 20B tokens, validation perplexityvalidation perplexity: 16.23 -> 16.01Improved
increase expert granularity from n=1024 to n=256 at constant FLOPs (ablation)MoE · 1.4B MoE, 50B tokens, validation perplexityvalidation perplexity: 13.38 -> 13.32Improved
make experts finer (n=256 to n=64) at constant FLOPs (ablation)MoE · 1.4B MoE, 50B tokens, validation perplexityvalidation perplexity: 13.32 -> 13.50Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy