Research map / Architectures and efficiency / Mixture of experts
Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism
Chenwei Cui, Rockwell Jackson, Benjamin Joseph Herrera, Ana M. Tárano, Hannah Kerner
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Multi-Head LatentMoE with Head Parallel instead of MoE with Expert Parallel (4.2B total) | MoE · FineWeb-EDU 50B tokens, 0.2B-4.2B | training hours: 316.34 -> 174.07 | Improved |
| use Multi-Head LatentMoE with Head Parallel instead of MoE EP (2.2B total) | MoE · FineWeb-EDU 50B tokens, 0.2B-2.2B | training hours: 177.09 -> 157.94 | Improved |
| use Multi-Head LatentMoE HP instead of MoE EP | MoE · zero-shot average, 0.2B-4.2B | average accuracy: 50.00 -> 51.29 | Improved |
| double granularity (G) on Multi-Head LatentMoE HP (ablation) | MH LatentMoE HP · 0.2B-4.2B | average accuracy: 51.29 -> 52.49 | Improved |
| double granularity on Multi-Head LatentMoE HP (ablation) | MH LatentMoE HP · 0.2B-4.2B | training hours: 174.07 -> 293.45 | Hurt |
| use Head Parallel instead of Expert Parallel under skewed load | Expert Parallel · 4 GPUs, Zipf skew 0 to 2 | all-to-all latency and VRAM | Improved |
| use IO-aware routing instead of naive torch.matmul routing | Multi-Head LatentMoE · B=40, T=2048, Nh=8, dh=128 | memory footprint | Improved |
| use FlexAttention-based expert computation instead of grouped GEMM | Multi-Head LatentMoE · profiling, 4 H100 | expert computation latency: 35.45 ms -> 18.40 ms | Improved |
| multi-head routing overhead vs standard routing | Multi-Head LatentMoE · profiling, 4 H100 | routing latency: 1.72 ms -> 6.52 ms | Hurt |
| use 4 x 256 heads instead of 8 x 128 (ablation) | Multi-Head LatentMoE · FineWeb-EDU 2.5B tokens | validation loss: 3.48 -> 3.41 | Mixed |
| use 16 x 64 heads instead of 8 x 128 (ablation) | Multi-Head LatentMoE · FineWeb-EDU 2.5B tokens | validation loss: 3.48 -> 3.56 | Hurt |
| add separate routing tokens for routing decisions (ablation) | Multi-Head LatentMoE · FineWeb-EDU 2.5B tokens | validation loss | No clear effect |
Closest papers by meaning
- FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training (NeurIPS 2025 · similarity 0.91)
- Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts (ICML 2025 · similarity 0.90)
- Occult: Optimizing Collaborative Communications across Experts for Accelerated Parallel MoE Training and Inference (ICML 2025 · similarity 0.90)
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.89)
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.88)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
