Research map / Architectures and efficiency / Mixture of experts
MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoE++, a heterogeneous MoE that adds zero, copy, and constant "zero-computation" experts alongside FFN experts plus gating residuals, trained from scratch on public data at 0.6B to 7B scale.
Problem
Vanilla MoE activates a fixed number of FFN experts for every token regardless of difficulty, causing training and inference inefficiency and sub-optimal performance, and expert communication/load imbalance across GPUs.
Approach
Add three near-parameter-free zero-computation experts (zero: discard; copy: skip via identity; constant: replace with a trainable vector mixed with input) so simple tokens use fewer FFN experts, and add gating residuals that carry router scores from the previous layer, plus a heterogeneous load-balance loss and capacity allocation.
Key results
- MoE++ 1B/(16+4)E at tau=0.75 averages 47.7 vs 46.6 for vanilla MoE 1B/16E on the ablation suite, with 15.2% throughput increase.
- MoE++ 7B/(16+4)E (1T tokens) averages 56.2 on the Table of equivalent-activated-parameter comparisons, outperforming OpenMoE-8B/32E on the shared tasks while using about 57% of its compute.
- Expert forward throughput increases of 15% to 111% over vanilla MoE across 0.6B to 7B settings, depending on tau.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace vanilla MoE with MoE++ using zero-computation experts and tau 0.75 | 7B/16E MoE · PIQA | accuracy: 72.6 -> 73.6 | Helped |
| replace vanilla MoE with MoE++ using zero-computation experts and tau 0.75 | 7B/16E MoE · WinoGrande | accuracy: 58.8 -> 58.2 | Hurt |
| replace vanilla MoE with MoE++ using zero-computation experts and tau 0.75 | 7B/16E MoE · 7B model, 100B tokens | throughput increase: 27.8% | Helped |
| add all three zero-computation expert types (zero, copy, and constant) (ablation) | MoE++ 1B/(16+4)E · PIQA | accuracy: 68.4 -> 70.3 | Helped |
| add a constant expert (ablation) | MoE++ 1B/(16+4)E · Average | accuracy: 46.6 -> 47.4 | Helped |
| add gating residuals (ablation) | MoE++ 1B/(16+4)E · Average | accuracy: 47.5 -> 47.7 | Helped |
| lower tau from 0.75 to 0.10 (ablation) | MoE++ 0.6B/(8+4)E · 0.6B model, 100B tokens | throughput increase: 25.2% -> 164.5% | Helped |
| lower tau from 0.75 to 0.10 (ablation) | MoE++ 0.6B/(8+4)E · Average | accuracy: 45.6 -> 43.4 | Hurt |
Surprising findings
- Constant experts contribute more to performance than zero or copy experts in the ablation.
- Smaller tau (more tokens routed to zero-computation experts) yields higher throughput but generally lower accuracy on some tasks, with NQ and LAMBADA showing mixed trends.
Limitations
- Scale limited to 7B parameters due to compute.
- Heterogeneity only within a single MoE layer; cross-layer heterogeneity left to future work.
- Only FFN layers replaced; attention not integrated with MoE++.
Experimental setup
Datasets: RedPajama Dolma Pile
Benchmarks: SciQ PIQA WinoGrande ARC-E ARC-C HellaSwag LogiQA BoolQ LAMBADA NQ MMLU
Metrics: accuracy exact match expert forward throughput average accuracy
Baselines: Vanilla MoE (0.6B/8E, 1B/16E, 2B/32E, 7B/16E) OpenMoE-8B/32E LLaMA2-7B OPT Pythia TinyLlama-1.1B Open-LLaMA-3B Sheared-LLaMA LLaMA-MoE-3.0B
Builds on: Switch Transformer GShard Expert Choice DeepSeekMoE Megatron
Closest papers by meaning
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.92)
- Mixture of Lookup Experts (ICML 2025 · similarity 0.91)
- Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts (ICML 2025 · similarity 0.90)
- LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing (NeurIPS 2024 · similarity 0.90)
- FlashMoE: Fast Distributed MoE in a Single Kernel (NeurIPS 2025 · similarity 0.90)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.