AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism

Chenwei Cui, Rockwell Jackson, Benjamin Joseph Herrera, Ana M. Tárano, Hannah Kerner

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use Multi-Head LatentMoE with Head Parallel instead of MoE with Expert Parallel (4.2B total)MoE · FineWeb-EDU 50B tokens, 0.2B-4.2Btraining hours: 316.34 -> 174.07Improved
use Multi-Head LatentMoE with Head Parallel instead of MoE EP (2.2B total)MoE · FineWeb-EDU 50B tokens, 0.2B-2.2Btraining hours: 177.09 -> 157.94Improved
use Multi-Head LatentMoE HP instead of MoE EPMoE · zero-shot average, 0.2B-4.2Baverage accuracy: 50.00 -> 51.29Improved
double granularity (G) on Multi-Head LatentMoE HP (ablation)MH LatentMoE HP · 0.2B-4.2Baverage accuracy: 51.29 -> 52.49Improved
double granularity on Multi-Head LatentMoE HP (ablation)MH LatentMoE HP · 0.2B-4.2Btraining hours: 174.07 -> 293.45Hurt
use Head Parallel instead of Expert Parallel under skewed loadExpert Parallel · 4 GPUs, Zipf skew 0 to 2all-to-all latency and VRAMImproved
use IO-aware routing instead of naive torch.matmul routingMulti-Head LatentMoE · B=40, T=2048, Nh=8, dh=128memory footprintImproved
use FlexAttention-based expert computation instead of grouped GEMMMulti-Head LatentMoE · profiling, 4 H100expert computation latency: 35.45 ms -> 18.40 msImproved
multi-head routing overhead vs standard routingMulti-Head LatentMoE · profiling, 4 H100routing latency: 1.72 ms -> 6.52 msHurt
use 4 x 256 heads instead of 8 x 128 (ablation)Multi-Head LatentMoE · FineWeb-EDU 2.5B tokensvalidation loss: 3.48 -> 3.41Mixed
use 16 x 64 heads instead of 8 x 128 (ablation)Multi-Head LatentMoE · FineWeb-EDU 2.5B tokensvalidation loss: 3.48 -> 3.56Hurt
add separate routing tokens for routing decisions (ablation)Multi-Head LatentMoE · FineWeb-EDU 2.5B tokensvalidation lossNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy