AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

Jingze Shi, Zhangyang Peng, Yizhang Zhu, Yifan Wu, Guang Liu, Yuyu Luo

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add shared dense MLP branch (ablation)OmniMoE · Ablation, relative to fullPPL / reasoning performance: PPL 1.2x without it; reasoning 0.79x without itImproved
add shared dense MLP branch (ablation)OmniMoE · Ablation, relative to fulllatency: 0.86x latency without itHurt
use Cartesian Product Router instead of dense routing projection (ablation)OmniMoE · Ablation, relative to fulllatency / memory: 30.6x latency and 337.5x memory without itImproved
use Cartesian Product Router instead of dense routing projection (ablation)OmniMoE · Ablation, relative to fullexpert usage / unevenness: expert usage 100% vs 4%; unevenness 0.24 vs 0.77Improved
use Cartesian Product Router instead of dense routing projection (ablation)OmniMoE · Ablation, relative to fullPPL / knowledge performance: PPL 1.4x, knowledge 0.66x without itImproved
add Expert-Centric Scheduling (ablation)OmniMoE · Ablation, relative to fulllatency / memory: 24.8x latency and 417.7x memory without itImproved
add Expert-Centric Scheduling (ablation)OmniMoE · Ablation, relative to fullPPL and downstream quality: 1.0x on all quality metricsNo clear effect
replace Gshard/DeepSeekMoE/PEER FFN with OmniMoEMoE FFN · 7 zero-shot benchmarks (6.4B-A1.7B)average accuracy: 50.9 vs 50.2 (DeepSeekMoE), 48.9 (PEER)Improved
replace PEER with OmniMoEMoE FFN · Reasoning benchmarks (6.4B-A1.7B)average accuracy: 27.0 -> 42.5Improved
replace PEER with OmniMoEfine-grained MoE · Inference latency at 4,096 tokenslatency: 73 ms -> 6.7 msImproved
replace Gshard/DeepSeekMoE/PEER with OmniMoEMoE FFN · Validation perplexity scalingperplexityImproved
replace DeepSeekMoE with OmniMoEMoE FFN · 7 zero-shot benchmarks (6.4B-A1.7B)HellaSwag accuracy: 61.2 -> 60.9Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy