Research map / Architectures and efficiency / Mixture of experts
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add sparse MoE experts to the 1.3B dense model | 1.3B NLG model · LAMBADA | accuracy: 63.65 -> 69.84 | Improved |
| Add sparse MoE experts to the 1.3B dense model | 1.3B NLG model · TriviaQA | accuracy: 10.05 -> 31.29 | Improved |
| Replace standard MoE with pyramid-residual MoE | 1.3B+MoE-128 · BoolQ | accuracy: 64.92 -> 67.16 | Improved |
| Replace standard MoE with pyramid-residual MoE | 1.3B+MoE-128 · TriviaQA | accuracy: 31.29 -> 28.86 | Hurt |
| Reduce PR-MoE depth from 24 to 21 layers without distillation (ablation) | 350M PR-MoE · BoolQ | accuracy: 59.88 -> 52.35 | Hurt |
| Add staged knowledge distillation to the 21-layer student (ablation) | 350M PR-MoE · BoolQ | accuracy: 52.35 -> 58.07 | Improved |
| Apply knowledge distillation throughout training rather than stopping it early (ablation) | 350M PR-MoE student · LAMBADA | accuracy: 63.46 vs. 61.56 | Hurt |
| Apply knowledge distillation throughout training rather than stopping it early (ablation) | 1.3B PR-MoE student · BoolQ | accuracy: 65.66 vs. 64.16 | Hurt |
| Replace top-2 expert routing with a fixed dense branch plus one selected expert (ablation) | 350M MoE · Validation loss and training speed | training speed: more than 10% | Improved |
| Use residual-MoE rather than top-2 routing (ablation) | 350M MoE · Validation loss | validation loss | No clear effect |
| Use staged KD rather than KD for the entire pre-training process (ablation) | PR-MoE student · Pre-training validation loss | validation loss | Improved |
| Combine pyramid-MoE and residual-MoE (ablation) | 350M MoE · Validation loss | validation loss: around 0.01 | Improved |
Closest papers by meaning
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.92)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.91)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.91)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.91)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.91)
- BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
