Research map / Architectures and efficiency / Mixture of experts
Sparse MoE as regularization and attention
Applies sparse expert layers to attention, dropout-style regularization, and sparsely activated transformers.
45 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Mixture of experts, keyword cluster “sparse · scaling · moes”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
11 papers in 2023–24 and 24 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 28.2% to 10.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (ICLR 2017 · 261 citations)
- 2020 · Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts (NeurIPS 2020 · 23 citations)
- 2021 · GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (ICLR 2021 · 351 citations)
- 2022 · Go Wider Instead of Deeper (AAAI 2022 · 55 citations)
- 2023 · Mod-Squad: Designing Mixtures of Experts As Modular Multi-Task Learners (CVPR 2023 · 90 citations)
- 2024 · SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · 8 citations)
- 2025 · Rethinking U-Net: Task-Adaptive Mixture of Skip Connections for Enhanced Medical Image Segmentation (AAAI 2025 · 8 citations)
- 2026 · SpanNorm: Reconciling Training Stability and Performance in Deep Transformers (ICML 2026 · 0 citations)
Earliest papers here
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (ICLR 2017 · 261 citations)
- Hard Mixtures of Experts for Large Scale Weakly Supervised Vision (CVPR 2017 · 74 citations)
- Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts (NeurIPS 2020 · 23 citations)
What has been tried
35 papers here report 416 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 6%, hurt 23%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Replace U-Net skip connections with TA-MoSC and add BEU | GlaS · Dice (%) | 85.45±1.26 -> 92.00±0.30 | Helped | Rethinking U-Net: Task-Adaptive Mixture of Skip Connections for Enhanced Medical Image Segmentation (AAAI 2025 · 8 citations) |
| Share expert selection between source and destination | C4 · perplexity | 22.53 -> 22.81 | Hurt | SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · 8 citations) |
| remove bagging from experts and gates | MultiMNIST · Mean ACC | Hurt | Bagging-Expert Network for Multi-Task Learning: A Depolarization Solution in Multi-Gate Mixture-of-Experts (AAAI 2025 · 3 citations) | |
| Add shared heads | ImageNet-1K, 100 epochs, 75% heads · accuracy | 75.6 -> 78.3 | Helped | MoH: Multi-Head Attention as Mixture-of-Head Attention (ICML 2025 · 3 citations) |
| use hierarchical decoder gating (MVMoE/4E-L) instead of base input-choice gating | trained VRPs · performance | Mixed | MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Experts (ICML 2024 · 3 citations) | |
| increase expert counts beyond E1=[4, 8, 12, 16], E2=[8, 16, 24, 32] | PANORAMA · mDSC (%) | 77.5 -> 76.3; 77.5 -> 75.8; 77.5 -> 77.5 | Mixed | Mamba Goes HoME: Hierarchical Soft Mixture-of-Experts for 3D Medical Image Segmentation (NeurIPS 2025 · 2 citations) |
| Remove the direct pathway in MHA | Base-capability evaluations · base capability | Hurt | How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider and MoE Transformers (NeurIPS 2024 · 2 citations) | |
| remove ACT | C4, 44M · PPL | 40.50 -> 21.51 | Helped | MoEUT: Mixture-of-Experts Universal Transformers (NeurIPS 2024 · 2 citations) |
| Add HDBSCAN clustering to random expert extraction | ImageNet-1k · Top-1 accuracy | 58.64 -> 75.36 | Helped | Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks (CVPR 2025 · 1 citation) |
| add CaPFN bootstrapped adapter finetuning | 10 shared ablation datasets · mean rank | 5.10 ± 1.12 -> 2.75 ± 1.94 | Helped | Mixture of In-Context Prompters for Tabular PFNs (ICLR 2025 · 1 citation) |
| use Expert Race instead of Token Choice | ImageNet · FID | 9.50 -> 8.03 | Helped | Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts (ICML 2025 · 0 citations) |
| Add the three-stage training mechanism | 80 tasks · normalized score | 71.90± 0.70 -> 76.67± 0.29 | Helped | Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 17 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1k · 3 of 17 papers
- CIFAR-100 · 3 of 17 papers
- RedPajama · 2 of 17 papers
- CIFAR-10 · 2 of 17 papers
Baselines
- U-Net · 2 of 17 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 35 papers; from OpenAlex and the papers' affiliations).
- Peking University · 3 papers
- Google · 2 papers
- Georgia Institute of Technology · 2 papers
- Tsinghua University · 2 papers
- Nanyang Technological University · 2 papers
- National University of Singapore · 2 papers
- IDSIA · 2 papers
- Harvard University · 2 papers
Use it
Download BibTeX of 33 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Expert routing and specialization (150 papers)
- Efficient MoE language model systems (72 papers)
Elsewhere in the topic
- Mixture-of-experts LoRA (18 papers)
- MoE LLM compression (15 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).