Research map / Architectures and efficiency / Mixture of experts
Expert routing and specialization
Improves expert specialization, soft routing, and interpretation in mixture-of-experts models.
150 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2021–2026) · an approach in Mixture of experts, keyword cluster “sparse · scaling · moes”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
20 papers in 2023–24 and 121 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 51.3% to 50.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2021 · Scaling Vision with Sparse Mixture of Experts (NeurIPS 2021 · 29 citations)
- 2022 · Mixture-of-Experts with Expert Choice Routing (NeurIPS 2022 · 90 citations)
- 2023 · MoEC: Mixture of Expert Clusters (AAAI 2023 · 14 citations)
- 2024 · From Sparse to Soft Mixtures of Experts (ICLR 2024 · 25 citations)
- 2025 · Complexity Experts are Task-Discriminative Learners for Any Image Restoration (CVPR 2025 · 51 citations)
- 2026 · Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging (AAAI 2026 · 3 citations)
Earliest papers here
- Scaling Vision with Sparse Mixture of Experts (NeurIPS 2021 · 29 citations)
- DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (NeurIPS 2021 · 17 citations)
- Mixture-of-Experts with Expert Choice Routing (NeurIPS 2022 · 90 citations)
What has been tried
57 papers here report 612 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 8%, hurt 23%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| use complexity experts with complexity-biased routing | AIO-3 average · PSNR | 32.30 -> 32.57 | Helped | Complexity Experts are Task-Discriminative Learners for Any Image Restoration (CVPR 2025 · 51 citations) |
| Average expert outputs uniformly instead of learning combine weights | JFT-4B · precision-at-1 | 54.3% -> 53.6% | Hurt | From Sparse to Soft Mixtures of Experts (ICLR 2024 · 25 citations) |
| Replace parallel FFN experts with FME and add UaR | All-Weather average · PSNR | 28.95 -> 29.01 | Helped | Efficient Deweahter Mixture-of-Experts with Uncertainty-Aware Feature-Wise Linear Modulation (AAAI 2024 · 18 citations) |
| add expert buffering to dynamic gating | MT decoder, single-node · throughput | Hurt | Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · 15 citations) | |
| Increase expert count from 2 to 4 or 8 in MoDE | Tabular benchmark datasets · accuracy | MoDE accuracy is higher with 8 experts than 2 experts on all seven datasets | Helped | MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · 14 citations) |
| use 6 heads instead of 4 | English-focused language modeling · perplexity | 12.72 -> 12.41 | Helped | Multi-Head Mixture-of-Experts (NeurIPS 2024 · 6 citations) |
| freeze the expert-shared base FFN | Code generation · Overall | 61.0 -> 59.7 | Hurt | DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models (CVPR 2025 · 3 citations) |
| Increase granularity above G=1 | C4 language modeling · loss | Helped | Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · 3 citations) | |
| add checkpoint recycling to Soft MoE | CIFAR-100 · accuracy | 75.9 -> 84.7 | Helped | MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks (NeurIPS 2024 · 3 citations) |
| use Pos-3 instead of Pos-2 shortcut position | 8×A30-PCIe · training speedup | 1.43× -> 1.49× | Helped | Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts (ICML 2025 · 2 citations) |
| add load-balancing refinement to the L1 regularization | Downstream tasks, load balancing ablation · Avg. accuracy | 39.52 -> 40.03 | Helped | ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing (ICLR 2025 · 2 citations) |
| add a larger MLP router instead of a linear router | Enwiki8 test · BPC | 1.128 -> 1.137 | Hurt | Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · 2 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| MoE-LLaVA | MME | score | 1318 – 1431 | 2 papers, e.g. ICLR 2025 and ICLR 2025 |
What reviewers will expect
Datasets and baselines named in the 48 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- WikiText-103 · 7 of 48 papers
- FineWeb-Edu · 4 of 48 papers
- WikiText-2 · 3 of 48 papers
- C4 · 3 of 48 papers
- Winogrande · 3 of 48 papers
- MMLU · 3 of 48 papers
- LLaVA-mix-665k · 2 of 48 papers
- RedPajama-v2 · 2 of 48 papers
Baselines
- MC-SMoE · 3 of 48 papers
- Switch Transformer · 3 of 48 papers
- LLaVA-1.5 · 3 of 48 papers
- DeepSpeed-MoE · 2 of 48 papers
- M-SMoE · 2 of 48 papers
- Loss-Free Balancing · 2 of 48 papers
- NAEE · 2 of 48 papers
- XMoE · 2 of 48 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 141 papers; from OpenAlex and the papers' affiliations).
- Peking University · 5 papers
- IDEAS NCBR · 4 papers
- The Chinese University of Hong Kong · 3 papers
- Microsoft · 3 papers
- Nomagic · 3 papers
- University of Warsaw · 3 papers
- IDEAS NCBR Sp · 3 papers
- National University of Singapore · 3 papers
- Hong Kong University of Science and Technology · 3 papers
- Nanjing University · 2 papers
Use it
Download BibTeX of 30 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Sparse MoE as regularization and attention (45 papers)
- Efficient MoE language model systems (72 papers)
Elsewhere in the topic
- Mixture-of-experts LoRA (18 papers)
- MoE LLM compression (15 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).