Research map / Architectures and efficiency / Mixture of experts
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Jingze Shi, Zhangyang Peng, Yizhang Zhu, Yifan Wu, Guang Liu, Yuyu Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add shared dense MLP branch (ablation) | OmniMoE · Ablation, relative to full | PPL / reasoning performance: PPL 1.2x without it; reasoning 0.79x without it | Improved |
| add shared dense MLP branch (ablation) | OmniMoE · Ablation, relative to full | latency: 0.86x latency without it | Hurt |
| use Cartesian Product Router instead of dense routing projection (ablation) | OmniMoE · Ablation, relative to full | latency / memory: 30.6x latency and 337.5x memory without it | Improved |
| use Cartesian Product Router instead of dense routing projection (ablation) | OmniMoE · Ablation, relative to full | expert usage / unevenness: expert usage 100% vs 4%; unevenness 0.24 vs 0.77 | Improved |
| use Cartesian Product Router instead of dense routing projection (ablation) | OmniMoE · Ablation, relative to full | PPL / knowledge performance: PPL 1.4x, knowledge 0.66x without it | Improved |
| add Expert-Centric Scheduling (ablation) | OmniMoE · Ablation, relative to full | latency / memory: 24.8x latency and 417.7x memory without it | Improved |
| add Expert-Centric Scheduling (ablation) | OmniMoE · Ablation, relative to full | PPL and downstream quality: 1.0x on all quality metrics | No clear effect |
| replace Gshard/DeepSeekMoE/PEER FFN with OmniMoE | MoE FFN · 7 zero-shot benchmarks (6.4B-A1.7B) | average accuracy: 50.9 vs 50.2 (DeepSeekMoE), 48.9 (PEER) | Improved |
| replace PEER with OmniMoE | MoE FFN · Reasoning benchmarks (6.4B-A1.7B) | average accuracy: 27.0 -> 42.5 | Improved |
| replace PEER with OmniMoE | fine-grained MoE · Inference latency at 4,096 tokens | latency: 73 ms -> 6.7 ms | Improved |
| replace Gshard/DeepSeekMoE/PEER with OmniMoE | MoE FFN · Validation perplexity scaling | perplexity | Improved |
| replace DeepSeekMoE with OmniMoE | MoE FFN · 7 zero-shot benchmarks (6.4B-A1.7B) | HellaSwag accuracy: 61.2 -> 60.9 | Hurt |
Closest papers by meaning
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.89)
- Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts (ICML 2025 · similarity 0.88)
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.87)
- MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenario (NeurIPS 2026 · similarity 0.87)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.87)
- FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
