Research map / Architectures and efficiency
Mixture of experts: research map
318 papers on Mixture of experts in Architectures and efficiency, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 2 clusters and 5 approaches. The busiest year so far is 2026.
What it is
A mixture-of-experts (MoE) model has many expert sub-networks and a router that sends each token to only a few of them, so the model can have many parameters while each input uses little computation. Research covers routing, expert specialisation, compression and efficient systems.
Terms you will see on this page:
- Expert
- one of the sub-networks inside an MoE layer.
- Router
- the small network that picks which experts process each token.
- Sparse activation
- only part of the model running for each input.
Start reading with the most cited papers here:
- Expert Gate: Lifelong Learning With a Network of Experts (CVPR 2017 · 613 citations)
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (ICLR 2021 · 351 citations)
Within Architectures and efficiency, its share grew from 2.0% in 2023–24 to 6.7% in 2025–26 (39 → 239 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Mixture of experts in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Efficient MoE language model systems rising: 8% → 28% of the topic (3 → 67 papers)
- Sparse MoE as regularization and attention falling: 28% → 10% of the topic (11 → 24 papers)
Papers from 2025–2026 that are already cited most:
- Complexity Experts are Task-Discriminative Learners for Any Image Restoration (CVPR 2025 · 51 citations)
- SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Tracking (CVPR 2025 · 16 citations)
- Rethinking U-Net: Task-Adaptive Mixture of Skip Connections for Enhanced Medical Image Segmentation (AAAI 2025 · 8 citations)
- EWMoE: An Effective Model for Global Weather Forecasting with Mixture-of-Experts (AAAI 2025 · 6 citations)
Approaches and key papers
Routing and sparse MoE models · 283 papers
Keywords: sparse, scaling, moes
Approaches in this cluster:
- Expert routing and specialization (156 papers)
Improves expert specialization, soft routing, and interpretation in mixture-of-experts models. - Sparse MoE as regularization and attention (54 papers)
Applies sparse expert layers to attention, dropout-style regularization, and sparsely activated transformers. - Efficient MoE language model systems (73 papers)
Restructures LLMs into MoE and speeds distributed MoE training with routing and scheduling.
Most cited and most cited since 2024:
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (ICLR 2021 · 351 citations)
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (ICLR 2017 · 261 citations)
- Complexity Experts are Task-Discriminative Learners for Any Image Restoration (CVPR 2025 · 51 citations)
- From Sparse to Soft Mixtures of Experts (ICLR 2024 · 25 citations)
MoE with LoRA and compression · 35 papers
Keywords: lora, rank, tuning
Approaches in this cluster:
- Mixture-of-experts LoRA (20 papers)
Combines low-rank adapters with expert routing for multi-task and parameter-efficient adaptation. - MoE LLM compression (15 papers)
Compresses mixture-of-experts LLMs through decomposition, delta decompression, and expert-level quantization.
Most cited and most cited since 2024:
- Expert Gate: Lifelong Learning With a Network of Experts (CVPR 2017 · 613 citations)
- Multi-Task Dense Prediction via Mixture of Low-Rank Experts (CVPR 2024 · 50 citations)
- SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Tracking (CVPR 2025 · 16 citations)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · 11 citations)
Related topics in Architectures and efficiency
- Pruning and efficient tuning (1,335)
- Attention and transformers (2,598)
- Neural architecture search (548)
- Quantization (785)
- RNNs and normalization (3,071)
- Spiking networks (333)
- CNN design (1,051)
