Research map / Architectures and efficiency / Mixture of experts
Efficient MoE language model systems
Restructures LLMs into MoE and speeds distributed MoE training with routing and scheduling.
72 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2021–2026) · an approach in Mixture of experts, keyword cluster “sparse · scaling · moes”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
3 papers in 2023–24 and 67 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 7.7% to 28.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2021 · BASE Layers: Simplifying Training of Large, Sparse Models (ICML 2021 · 63 citations)
- 2022 · GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (ICML 2022 · 165 citations)
- 2023 · SCoMoE: Efficient Mixtures of Experts with Structured Communication (ICLR 2023)
- 2024 · $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · 0 citations)
- 2025 · OLMoE: Open Mixture-of-Experts Language Models (ICLR 2025 · 7 citations)
- 2026 · EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models (AAAI 2026 · 1 citation)
Earliest papers here
- BASE Layers: Simplifying Training of Large, Sparse Models (ICML 2021 · 63 citations)
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (ICML 2022 · 165 citations)
- SCoMoE: Efficient Mixtures of Experts with Structured Communication (ICLR 2023)
What has been tried
26 papers here report 298 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 7%, hurt 22%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| fine-grained routing: use 32 experts with 4 activated instead of 8 experts with 1 activated | HellaSwag and MMLU Var · downstream performance | around 10% | Helped | OLMoE: Open Mixture-of-Experts Language Models (ICLR 2025 · 7 citations) |
| Add 3D absolute position embedding instead of standard ViT position embedding | ERA5 weather forecasting, all forecast times · forecast performance | Helped | EWMoE: An Effective Model for Global Weather Forecasting with Mixture-of-Experts (AAAI 2025 · 6 citations) | |
| use last-token routing weights instead of averaging routing weights across tokens | STS12-16 average · Spearman correlation | 46.03 -> 61.18 | Helped | Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (ICLR 2025 · 2 citations) |
| remove router bias | Four-set evaluation (24 tasks) · average | 46.7 -> 45.8 | Hurt | FlexOLMo: Open Language Models for Flexible Data Use (NeurIPS 2025 · 1 citation) |
| Remove math and code training data | 12 high-resource languages · average accuracy | 48.6 -> 42.7 | Hurt | Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts (ICLR 2025 · 1 citation) |
| add AGQ to GPTQ | 4-bit, seven-task average · avg. accuracy | 39.01 -> 39.50 | Helped | MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · 0 citations) |
| Use expert outputs rather than router-logits for hierarchical clustering | Qwen 45x2.7B, zero-shot · Average accuracy | 0.3153 -> 0.5459 | Helped | Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025 · 0 citations) |
| Add router jitter noise | Pretraining · pretraining loss | No clear effect | Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · 0 citations) | |
| Add balanced expert routing without confidence-aware selection | 8-task zero-shot average · accuracy (%) | 62.2 -> 57.9 | Hurt | CryptoMoE: Privacy-Preserving and Scalable Mixture of Experts Inference via Balanced Expert Routing (NeurIPS 2025 · 0 citations) |
| Increase MoE sparsity by reducing activated experts per token K | Varied batch sizes · speedup | Mixed | MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · 0 citations) | |
| add MHA computing and gating pipelining | customized MoE layer, 16 GPUs · iteration time | 1115.2 -> 1012.6 | Helped | FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training (NeurIPS 2025 · 0 citations) |
| Expertize the smaller dt and x projections rather than sharing their parameters | SlimPajama validation, context length 16384 · perplexity | 8.85 -> 8.87 | Hurt | Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 38 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- C4 · 7 of 38 papers
- MMLU · 4 of 38 papers
- HellaSwag · 4 of 38 papers
- Wikipedia · 3 of 38 papers
- WikiText2 · 3 of 38 papers
- MT-bench · 3 of 38 papers
- StarCoder · 3 of 38 papers
- ARC-easy · 3 of 38 papers
Baselines
- ScheMoE · 2 of 38 papers
- Tutel · 2 of 38 papers
- Model soup · 2 of 38 papers
- BTX · 2 of 38 papers
- Top-K routing · 2 of 38 papers
- Mixtral-Offloading · 2 of 38 papers
- llama.cpp · 2 of 38 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 70 papers; from OpenAlex and the papers' affiliations).
- University of Science and Technology of China · 3 papers
- Shanghai Jiao Tong University · 3 papers
- Huawei · 2 papers
- University of Texas at Austin · 2 papers
- Peking University · 2 papers
- ByteDance · 2 papers
- Massachusetts Institute of Technology · 2 papers
- Qualcomm · 2 papers
- Google · 2 papers
- Meta · 2 papers
Use it
Download BibTeX of 22 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Expert routing and specialization (150 papers)
- Sparse MoE as regularization and attention (45 papers)
Elsewhere in the topic
- Mixture-of-experts LoRA (18 papers)
- MoE LLM compression (15 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).