Research map / Architectures and efficiency / Mixture of experts
Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ScMoE, a shortcut-connected MoE architecture (top-1 MoE on preceding-layer representations plus a shared expert on current-layer representations) with an adaptive overlap scheduler that hides All-to-All communication behind computation in expert parallelism.
Problem
All-to-All communication in expert-parallel MoE training and inference is a major bottleneck (up to ~50% of MoE time), and existing pipelining and hierarchical optimizations remain limited by sequential dependencies between communication and computation.
Approach
Reroute the MoE input through a shortcut from the preceding Transformer layer so the routed-expert path (with its All-to-All) is decoupled from the current layer's shared-expert computation, enabling communication to overlap with attention and MLP computation; an adaptive scheduler picks where to place expert computation in the stream.
Key results
- On SwinV2-MoE-S in 8xA30-PCIe, ScMoE (Pos-3) achieves 1.49x training and 1.82x inference speedup over top-2 MoE, with Pos-2 at 79.38% top-1 accuracy vs 79.33% for top-2.
- On 8xA800-NVLink, ScMoE reaches 1.12x training and 1.17x inference speedup on GPT2-MoE and 1.14x/1.21x on LLaMA2-MoE, with average zero-shot scores of 38.69 and 38.96 vs 37.86 and 38.39 for top-2.
- Expert offloading built on ScMoE reduces peak GPU memory by up to 60% (GPT3-MoE-XL) in single-A30 inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard top-2 MoE with ScMoE Pos-2 | SwinV2-MoE-S · ImageNet-1K test | Acc@1: 79.33% -> 79.38% | No clear effect |
| replace standard top-2 MoE with ScMoE Pos-2 | SwinV2-MoE-S · 8×A30-PCIe | training speedup: 1 -> 1.43× | Helped |
| replace standard top-2 MoE with ScMoE Pos-2 | SwinV2-MoE-S · 8×A30-PCIe | inference speedup: 1 -> 1.66× | Helped |
| use Pos-3 instead of Pos-2 shortcut position (ablation) | ScMoE · 8×A30-PCIe | training speedup: 1.43× -> 1.49× | Helped |
| use Pos-3 instead of Pos-2 shortcut position (ablation) | ScMoE · 8×A30-PCIe | inference speedup: 1.66× -> 1.82× | Helped |
| use CG-1 coefficient gating instead of Direct Add (ablation) | ScMoE Pos-2 · GPT-2 MoE final validation | loss: 3.236811 -> 3.224763 | Helped |
| use CG-1 coefficient gating instead of CG-2 (ablation) | ScMoE Pos-2 · GPT-2 MoE final validation | loss: 3.232943 -> 3.224763 | Helped |
| use Pos-2 instead of Pos-1 shortcut position (ablation) | ScMoE with CG-1 · GPT-2 MoE final validation | loss: 3.237530 -> 3.224763 | Helped |
| use Pos-2 instead of Pos-3 shortcut position (ablation) | ScMoE with CG-1 · GPT-2 MoE final validation | loss: 3.241349 -> 3.224763 | Helped |
| add a second routed expert for preceding-layer representations (ablation) | ScMoE · GPT3-MoE-XL validation | perplexity: 16.46 -> 16.27 | Helped |
| replace standard top-2 MoE with ScMoE | GPT2-MoE · GPT2-MoE zero-shot tasks | AVG.: 37.86 -> 38.69 | Helped |
| replace standard top-2 MoE with ScMoE | LLaMA2-MoE · LLaMA2-MoE zero-shot tasks | AVG.: 38.39 -> 38.96 | Helped |
Surprising findings
- Shared-expert MoE gave the highest vision accuracy (79.53%) despite lower speedup than ScMoE, and DGMoE with a distinct-expert constraint matched top-2 quality.
- Preceding and current layer representations have cosine similarity close to 1.0, which the authors use to argue the shortcut preserves quality.
Limitations
- Gains shrink when communication is a small fraction of time (e.g., 1.12x on 8xA800-NVLink), and some configurations are slower than top-3 MoE baseline (0.94x).
- Experiments use relatively small models (up to 6.7B, 157M-scale vision) and limited hyperparameter tuning; the theoretical analysis is a gradient-propagation argument rather than a convergence proof.
- Hardware configurations are limited to A30 and A800 GPUs; the paper does not report results on other interconnects or larger scale deployments beyond 16 GPUs.
Experimental setup
Datasets: ImageNet-1K SlimPajama-627B (1B-token subset) OpenWebText WikiText-103
Benchmarks: ImageNet-1K HellaSwag PIQA WinoGrande BoolQ ARC-E OBQA RACE MathQA WikiText-103
Metrics: top-1 accuracy zero-shot accuracy validation loss perplexity training speedup inference speedup
Baselines: standard top-2 MoE standard top-1 MoE shared-expert MoE Tutel pipelined MoE Pre-gated MoE
Builds on: DeepSpeed-MoE shared expert Tutel pipelining for MoE all-to-all overlap Pre-gated MoE
Closest papers by meaning
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.90)
- Occult: Optimizing Collaborative Communications across Experts for Accelerated Parallel MoE Training and Inference (ICML 2025 · similarity 0.89)
- FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training (NeurIPS 2025 · similarity 0.89)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.89)
- FlashMoE: Fast Distributed MoE in a Single Kernel (NeurIPS 2025 · similarity 0.89)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.