Research map / Architectures and efficiency / Mixture of experts
SNaX: sparse narrow accelerated mixture of experts
Wentao Guo, Mayank Mishra, Xinle Cheng, Ion Stoica, Tri Dao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add token rounding routing instead of vanilla top-K token choice | MoE training (TR) · 1.8B MoE, 40B tokens, 8/256 activated, avg accuracy | Avg accuracy: 52.8 -> 53.5 | Improved |
| add token rounding instead of top-K token choice | MoE training (TR) · 1.4B MoE, 100B tokens, 2/128 activated, validation perplexity | validation perplexity: 13.32 -> 13.22 | Improved |
| add token rounding instead of top-K token choice | MoE training (TR) · 1.4B MoE, 100B tokens, 2/128 activated, avg accuracy | Avg accuracy: 52.0 -> 52.0 | No clear effect |
| use expert-choice routing instead of token rounding, then evaluate with TC | MoE training · 0.5B MoE, 40B tokens, validation perplexity | validation perplexity: 15.92 -> 18.61 | Hurt |
| fine-tune a TC router instead of using an auxiliary router for EC | EC routing · 0.5B MoE, 40B tokens, validation perplexity | validation perplexity: 21.80 -> 16.98 | Improved |
| add token rounding instead of token dropping | MoE training · 1.4B MoE, 100B tokens, avg accuracy | Avg accuracy: 51.3 -> 52.0 | Improved |
| always round expert frequency down (DOWN) instead of nearest rounding (ablation) | token rounding · 0.5B MoE, 40B tokens, validation perplexity | validation perplexity: 15.92 -> 16.10 | Hurt |
| always round expert frequency up instead of nearest rounding (ablation) | token rounding · 0.5B MoE, 40B tokens, validation perplexity | validation perplexity: 15.92 -> 15.89 | No clear effect |
| use stochastic rounding instead of nearest rounding (ablation) | token rounding · 0.5B MoE, 40B tokens, validation perplexity | validation perplexity: 15.92 -> 15.93 | No clear effect |
| increase expert granularity from n=1024 to n=256 at constant FLOPs (ablation) | MoE · 0.5B MoE, 20B tokens, validation perplexity | validation perplexity: 16.23 -> 16.01 | Improved |
| increase expert granularity from n=1024 to n=256 at constant FLOPs (ablation) | MoE · 1.4B MoE, 50B tokens, validation perplexity | validation perplexity: 13.38 -> 13.32 | Improved |
| make experts finer (n=256 to n=64) at constant FLOPs (ablation) | MoE · 1.4B MoE, 50B tokens, validation perplexity | validation perplexity: 13.32 -> 13.50 | Hurt |
Closest papers by meaning
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.90)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.90)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.89)
- Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts (ICML 2025 · similarity 0.89)
- LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing (NeurIPS 2024 · similarity 0.89)
- BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
