Research map / Architectures and efficiency / Mixture of experts
BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BigMac, a fine-grained MoE structure that projects tokens down before All-to-All communication and up afterward (DCCA), cutting All-to-All volume by 75% and speeding up training and inference.
Problem
All-to-All communication in fine-grained Mixture-of-Experts models becomes a severe bottleneck for training and inference, because fine-grained routing activates more experts and communicates at full hidden dimension.
Approach
Reorder the expert computation to descend-communicate-communicate-ascend (DCCA): place a down-projection before the first All-to-All and an up-projection after the second, so communication happens at a reduced dimension; the expert is redesigned as ascend-then-descend to keep parameter count comparable to fine-grained experts.
Key results
- BigMac reaches validation perplexity 13.69 in 22.8 hours versus 38.9 hours for the fine-grained MoE (1.71x faster) on GPT3-XL-based pre-training.
- BigMac reduces end-to-end training latency by up to 3.09x on Tutel and increases inference throughput by up to 3.11x on DeepSpeed-Inference.
- With r=0.25 and Top8 routing, All-to-All transfer size drops 75% (1,488 GB to 372 GB) at only 4.54% extra FLOPs and 1.35% extra parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Fine-Grained with BigMac | GPT3-XL MoE · PTB | perplexity: 51.0 -> 34.9 | Helped |
| Replace Fine-Grained with BigMac | GPT3-XL MoE · WikiText-103 | perplexity: 18.2 -> 16.8 | Helped |
| Replace Fine-Grained with BigMac | GPT3-XL MoE · LAMBADA | accuracy: 39.9 -> 40.8 | Helped |
| Replace Fine-Grained with BigMac | GPT3-Medium MoE · WikiText-103 | perplexity: 19.1 -> 19.1 | No clear effect |
| Replace Fine-Grained with BigMac | GPT3-Medium MoE · LAMBADA | accuracy: 38.3 -> 37.6 | Hurt |
| Replace Fine-Grained with BigMac | GPT3-Medium MoE · RACE-H | accuracy: 29.8 -> 30.7 | Helped |
| Replace Fine-Grained with BigMac | GPT3-XL MoE · training to validation perplexity 13.69 | training time: 38.9 hours -> 22.8 hours | Helped |
| Replace Fine-Grained with BigMac | MoE model · Megatron training, Top4 routing | end-to-end latency speedup: 1.53-2.41× | Helped |
| Replace Fine-Grained with BigMac | MoE model · Megatron training, Top8 routing | end-to-end latency speedup: 2.45-3.07× | Helped |
| Replace Fine-Grained with BigMac | MoE model · Megatron inference | throughput speedup: 1.72-2.45× | Helped |
| Replace Fine-Grained with BigMac | MoE model · Tutel training | training speedup: 1.71× to 3.09× | Helped |
| Replace Fine-Grained with BigMac | MoE model · Tutel inference | throughput speedup: 1.67-1.87× | Helped |
Surprising findings
- GPT-BigMac with Top8 routing still outperforms the fine-grained model with Top4 routing by 27.7-55.4% in end-to-end training latency.
- Under the same time budget, the conventional vanilla MoE fails to reach the target perplexity, while the fine-grained and BigMac structures do.
Limitations
- For small models without expert parallelism, BigMac slightly increases latency due to the extra projections.
- Downscaling factor r trades communication against model quality and was fixed at 0.25 for fair comparison, with no sweep reported.
- Model quality comparisons are on small GPT3-style models trained on modest token counts (3.6B and 14.8B).
Experimental setup
Datasets: Wikipedia OpenWebText2
Benchmarks: PTB WikiText-103 WikiText2 LAMBADA HellaSwag WinoGrande PIQA RACE-H GovReport NeedleInAHaystack
Metrics: perplexity accuracy recall end-to-end training latency inference throughput All-to-All latency
Baselines: GPT-Vanilla MoE GPT-Fine-Grained (DeepSeekMoE-style) ScheMoE Tutel DeepSpeed-Inference
Builds on: DeepSeekMoE GShard Megatron-LM Tutel DeepSpeed-Inference
Closest papers by meaning
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.90)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.90)
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.89)
- Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.