Research map / Architectures and efficiency / Mixture of experts
MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MoE-CAP, a benchmark and analysis framework for sparse MoE serving systems that jointly evaluates cost, accuracy and performance, with a CAP radar diagram and sparsity-aware S-MBU and S-MFU metrics.
Problem
Existing LLM benchmarks and utilization metrics (MBU, MFU) do not capture the cost-accuracy-performance trade-offs of sparse MoE systems, since they ignore selective expert activation and heterogeneous CPU/memory resources.
Approach
Categorize MoE systems by which two of cost, accuracy and performance they optimize (CAP trade-off), visualize them with a CAP radar diagram, and redefine MBU and MFU to count only activated experts (S-MBU, S-MFU).
Key results
- MoE-Infinity matches SGLang accuracy (91.1% EM on GSM8K) while cutting purchase cost by nearly 60%, at 2.6x higher latency (0.15 vs 0.058 s/token) on Qwen3-30B-A3B.
- Vanilla MBU overestimates memory access by over 260% for Mixtral-8x7B, while S-MBU matches profiled memory access with under 1% error.
- Full activation of DeepSeek-R1 requires about 18,901 GB/s bandwidth, while at batch size 1 the requirement drops to 1,040 GB/s, enabling consumer GPUs with offloading.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Offload experts instead of running the model with SGLang | Qwen3-30B-A3B inference on NVIDIA A5000 GPUs · GSM8K | decoding latency: 0.058 s/token vs. 0.15 s/token; 2.6× increase in latency | Hurt |
| Offload experts instead of running the model with SGLang | Qwen3-30B-A3B inference on NVIDIA A5000 GPUs · GSM8K | purchase cost: reduces purchase cost by nearly 60% | Helped |
| Use FP8 quantization rather than expert offloading | Qwen3-235B-A22B inference on NVIDIA H20 GPUs · Qwen3-235B-A22B | throughput: over 75× faster than MoE-Infinity | Helped |
| Use FP8 quantization rather than expert offloading | Qwen3-235B-A22B inference on NVIDIA H20 GPUs · Qwen3-235B-A22B | power cost: 2.7× higher power cost | Hurt |
| Account for selective expert activation in memory estimates | S-MBU compared with vanilla MBU on Mixtral-8x7B · GSM8K, batch sizes 1–64 | memory usage error: vanilla MBU ... over 260%; S-MBU ... less than 1% error | Helped |
| Increase batch size to 8 | MoE model inference · MATH | activated parameters: 53.05%, 46.79%, and 18.44% | Mixed |
Surprising findings
- At batch size 8, only 18.44% of DeepSeek-R1 parameters are activated, versus 53.05% for DeepSeek-V2-Lite, indicating sparsity varies strongly by model.
Limitations
- Covers only inference and decoding performance; prefilling, post-training and pre-training are not yet evaluated.
- Evaluation limited to a small set of MoE models, systems and four datasets.
Experimental setup
Datasets: MMLU GSM8K MATH Arena-Hard
Benchmarks: MMLU GSM8K MATH Arena-Hard
Metrics: exact match F1 win rate TPOT throughput S-MBU S-MFU power cost purchase cost
Baselines: SGLang K-Transformers MoE-Infinity SGLang-FP8 SGLang-AWQ vanilla MBU vanilla MFU
Builds on: MBU MFU MoE-Infinity Fiddler
Closest papers by meaning
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.91)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.91)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.91)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.90)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.