AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts

Xuan-Phi Nguyen, Shrey Pandit, Austin Xu, Caiming Xiong, Shafiq Joty

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
reroute excess tokens and expert weights to least-loaded GPUs (LLEP) instead of standard EPLLEP vs EP · MoE layer, extreme imbalance (95% into 1 expert)speedup: up to 6.11x (gpt-oss-120b layer)Improved
use LLEP instead of EP on peak memory per GPULLEP vs EP · MoE layer under imbalancepeak memory: EP up to 4x growth; LLEP near constantImproved
use LLEP for full-model forward throughputLLEP vs EP · gpt-oss-20b / gpt-oss-120b on Nemotron-Maththroughput: 2.2× / 1.88×Improved
use LLEP for full-parameter fine-tuningLLEP vs EP · gpt-oss-20b AIME'25wall-time to accuracy: 1.25× speedup, comparable accuracyImproved
use LLEP in pretraining DeepSeek-V3-scale model with bias load balancingLLEP vs EP · 600B pretraining from scratchspeedup: ~1.55×Improved
use LLEP instead of EPLB for prefillLLEP vs EPLB · gpt-oss-120b prefill, 32 GPUsthroughput over EP: 1.45× -> 1.94×Improved
add adaptive ratio lambda fallback to standard EP (ablation)LLEP · balanced routingspeedImproved
increase batch size (ablation)LLEP · speedup vs EPspeedupImproved
increase capacity factor alpha (ablation)LLEP · speedup vs EPspeedupHurt
use higher lambda at low imbalance (15-20%) with small batch (ablation)LLEP · speedup vs EP, B=8KspeedupImproved
increase hidden size D, H (ablation)LLEP · speedup vs EPspeedupImproved
apply LLEP in decode with small per-GPU batchLLEP · gpt-oss-120b decodespeedup: degrades to standard EPNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy