Research map / Architectures and efficiency / Mixture of experts
Revisiting Efficiency–Accuracy Scaling in Mixture-of-Experts Architectures
Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joseph Guman, Tiyasa Mitra, Ritchie Zhao, Ritika Borkar and 4 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale active experts K and total experts N by d/l with latent projection (l-MoEacc) | standard MoE · 95B-8BA, MMLU Pro | accuracy: 29.26 -> 34.91 | Improved |
| scale active experts K and total experts N by d/l with latent projection (l-MoEacc) | standard hybrid MoE · Hybrid-73B-8BA, MMLU Pro | accuracy: 48.30 -> 52.87 | Improved |
| latent compression without increasing K (l-MoEeff) (ablation) | standard MoE · 95B-8BA, Code | accuracy: 40.33 -> 40.68 | No clear effect |
| latent compression without increasing K (l-MoEeff) (ablation) | standard MoE · 95B-8BA, MMLU Pro | accuracy: 29.26 -> 34.75 | Improved |
| latent compression without increasing K (l-MoEeff) (ablation) | standard hybrid MoE · Hybrid-73B-8BA, Math | accuracy: 78.32 -> 77.01 | Hurt |
| raise compression ratio alpha above 4 (ablation) | l-MoEeff 16B-2B · 16B-2B validation loss | validation loss | Hurt |
| scale routed expert count N by alpha under 4x compression (ablation) | l-MoEeff 16B-2B · 16B-2B validation loss | validation loss | Improved |
| latent routing (l-MoEacc) vs standard MoE inference throughput at high concurrency | Hybrid-73B-8BA · H100 FP8 serving | throughput per GPU: 1725.9 -> 1625.8 at BS=128 | Hurt |
| use LatentMoE 1T instead of scaling standard MoE to 1.35T for the same accuracy | Kimi-K2 · projected serving Pareto frontier | slowdown of standard model: 1.24×–3.46× slower | Improved |
Closest papers by meaning
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.92)
- Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts (ICML 2025 · similarity 0.91)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.90)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.90)
- MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design (ICML 2025 · similarity 0.90)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
