Research map / Architectures and efficiency / Mixture of experts
Multi-Head Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-Head Mixture-of-Experts (MH-MoE), which splits each token into sub-tokens routed to different experts in parallel and merges them back, improving expert activation and fine-grained understanding at similar cost to sparse MoE.
Problem
Sparse Mixture of Experts models suffer from low expert activation (only a small subset of experts receive tokens) and lack fine-grained analysis of multiple semantic concepts within a single token.
Approach
A multi-head layer projects each token, then splits it into h sub-tokens that are routed independently to experts (top-k over sub-tokens), and a merge layer reintegrates them into the original token shape; this increases the number of activated experts per token without extra computation, with the expert inner dimension scaled by beta to match parameters.
Key results
- MH-MoE (8 experts) reduces English LM validation perplexity from 14.82 (X-MoE) to 12.72, and to 10.28 with 32 experts vs 11.96 for X-MoE.
- On VQAv2 test-dev, MH-MoE reaches 70.1 vs 68.4 for X-MoE with 8 experts.
- Expert activation rises to 90.71% in MH-MoE versus 8.33% in the SMoE example shown in the paper's motivation figure.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace X-MoE with MH-MoE | X-MoE · English-focused, multilingual, and masked multi-modal language modeling | perplexity: 14.82 -> 12.72; 11.96 -> 10.28; 7.19 -> 6.26; 6.02 -> 5.09; 16.34 -> 14.73; 12.68 -> 10.87 | Helped |
| replace X-MoE with MH-MoE | X-MoE · English language understanding, 8 and 32 experts | Avg accuracy: 38.7 -> 39.8; 39.1 -> 40.6 | Helped |
| replace X-MoE with MH-MoE | X-MoE · XNLI, 8 and 32 experts | Avg accuracy: 34.1 -> 34.7; 34.5 -> 35.3 | Helped |
| replace X-MoE with MH-MoE | X-MoE · VQAv2 test-dev | VQA score: 68.4 -> 70.1 | Helped |
| replace X-MoE with MH-MoE | X-MoE · NLVR2 dev and test-P | accuracy: 75.5 -> 77.0; 76.1 -> 77.8 | Helped |
| use 6 heads instead of 4 (ablation) | MH-MoE · English-focused language modeling | perplexity: 12.72 -> 12.41 | Helped |
| use 8 or 12 heads instead of 4 (ablation) | MH-MoE · English-focused language modeling | perplexity: 12.72 -> 12.95; 12.72 -> 13.28 | Hurt |
| remove token splitting-merging (TSM) (ablation) | MH-MoE · English-focused language modeling | perplexity: 12.72 -> 14.77 | Hurt |
| remove MLP layers (ablation) | MH-MoE · English-focused language modeling | perplexity: 12.72 -> 13.97 | Hurt |
| add MLP layers without TSM (ablation) | Dense and X-MoE · English-focused language modeling | perplexity: 16.23 -> 16.40; 14.82 -> 14.77 | Mixed |
| use 2 MLP layers instead of 1 (ablation) | MH-MoE · English-focused language modeling | perplexity: 12.72 -> 12.66 | Helped |
| use 3 MLP layers instead of 1 (ablation) | MH-MoE · English-focused language modeling | perplexity: 12.72 -> 12.87 | Hurt |
Surprising findings
- Adding token splitting without the MLP layers gives only a slight gain, and MLP alone gives no gain; both are needed together.
- Performance peaks at 6 heads and declines at higher head counts, since excessive token subdivision impairs semantics.
Limitations
- Gains are reported against a single SMoE backbone (X-MoE) and a dense baseline, with small-scale models (~360M params) in most experiments.
- Head-count sensitivity: performance drops for h above 6, so the hyperparameter must be tuned.
Experimental setup
Datasets: RedPajama C4 multilingual Wikipedia ImageNet-21K CC12M CC3M SBU COCO Visual Genome XNLI English Wikipedia BookCorpus
Benchmarks: LLM Evaluation Harness tasks (ARC, RTE, BookQA, Winogrande, PiQA, BoolQ, HellaSwag, TruthfulQA) XNLI VQAv2 NLVR2 COCO Captioning
Metrics: perplexity accuracy accuracy-normalization VQA score BLEU@4 METEOR CIDEr SPICE
Baselines: Dense Transformer X-MoE
Builds on: X-MoE GShard Switch Transformer BEiT v3
Closest papers by meaning
- Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025 · similarity 0.92)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.89)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.89)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.