Research map / Architectures and efficiency / Mixture of experts
Toward Efficient Inference for Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes MoE inference inefficiencies for language modeling and machine translation and proposes dynamic gating, expert buffering, and load balancing to raise throughput, cut memory, and balance expert load.
Problem
MoE Transformer inference is slow and memory-hungry due to static gating with fixed expert capacities, skewed expert activation, and limited GPU memory, with few direct baselines for efficient MoE serving.
Approach
Replace static capacity-padded gating with dynamic gating that sends variable token counts to experts (via a size all-to-all and index-based reordering), add an expert buffering cache that keeps hot experts in GPU memory and offloads the rest to CPU with LIFO eviction, and balance expert placement across GPUs using greedy and anti-correlation heuristics driven by runtime activation statistics.
Key results
- Dynamic gating improves maximum throughput by 6.21-11.55× for LM, 5.75-10.98× for MT encoder and 2.58-5.71× for MT decoder versus Fairseq static gating.
- Expert buffering reduces static GPU memory allocation by 1.47× while keeping latency increase nominal.
- Greedy load balancing increases throughput by up to 1.11× and 1.19× over pure dynamic gating; anti-correlation balancing gives about 1.02× on MT decoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace static gating with dynamic gating | MoE inference · LM | throughput versus Fairseq: 6.21× | Helped |
| replace static gating with dynamic gating | MoE inference · MT encoder | throughput versus Fairseq: 5.75× | Helped |
| replace static gating with dynamic gating | MoE inference · MT decoder | throughput versus Fairseq: 2.58× | Helped |
| replace static gating with dynamic gating | MoE inference · LM, batch size 8, Apple | activation memory: 6.29GB to 1.28GB; 79.6% | Helped |
| replace static gating with dynamic gating | MoE inference · MT, batch size 8, Apple | activation memory: 1.89GB to 1.05GB; 44.2% | Helped |
| add expert buffering to dynamic gating (ablation) | MoE inference · MT decoder, single-node | throughput | Hurt |
| add expert buffering to dynamic gating | MoE inference · MT decoder, two-node | throughput versus baselines: 2.21× | Helped |
| add expert buffering to dynamic gating | MoE inference · MT decoder, four-node | throughput versus baselines: 4.30× | Helped |
| buffer expert parameters in CPU memory | MoE inference · MT tasks, Apple | GPU static memory: 2.25GB | Helped |
| add Greedy load balancing | Dynamic Gating · LM | throughput: 1.11× and 1.19× | Helped |
| add Anti-Correlation load balancing | Dynamic Gating · MT decoder | throughput: 1.02× | Helped |
| use small batch sizes (4–16) rather than larger batches (ablation) | Dynamic Gating versus Megablock · LM and MT inference | throughput | Hurt |
Surprising findings
- Expert all-to-all communication is less significant to inference latency than gating and expert computation, unlike training.
- Dynamic gating underperforms Megablocks at small batch sizes (4-16) and only wins at larger batch sizes (32-80).
Limitations
- Results are on two clusters (Apple V100 up to 4 nodes and Pear A5000 single node) with specific LM and MT models; generalization to other scales is argued rather than shown.
- Expert buffering adds latency due to CPU-GPU transfers, with data rates peaking at 12 GB/s.
- Gains are smaller for small batch sizes and for correlated MT-decoder activations.
Experimental setup
Datasets: The Pile (Wikipedia, PubMed, GitHub) NLLB-200 validation data
Metrics: throughput latency memory use cache miss rate max load avg-max load
Baselines: Fairseq MoE (static gating) Tutel FasterMoE Megablocks
Builds on: GShard Fairseq MoE Tutel FasterMoE Megablocks
Closest papers by meaning
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.90)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.90)
- BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference (AAAI 2025 · similarity 0.90)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.