Research map / Architectures and efficiency / Mixture of experts
Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models
Jingcong Liang, Siyuan Wang, Miren Tian, Yitong Li, Duyu Tang, zhongyu wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the load-balancing loss (ablation) | toy MoE Baseline · toy models, m=16 | SRP: 43.56 -> 56.42 (load-balance SD 4.02 -> 13.21) | Improved |
| prioritize load balance more strongly in training (ablation) | toy MoE Baseline · toy models, m=16 | SRP: 43.56 -> 36.42 | Hurt |
| add shared experts (one or two) (ablation) | toy MoE Baseline · toy models, m=16 | SRP: 43.56 -> 41.38 (1 shared), 38.79 (2 shared) | Hurt |
| activate fewer experts per token (Top2) (ablation) | toy MoE Baseline · toy models, m=16 | SRP: 43.56 -> 27.13 | Hurt |
| activate more experts per token (Top16) (ablation) | toy MoE Baseline · toy models, m=16 | SRP: 43.56 -> 55.69 | Improved |
| use fewer total experts (32Exp) (ablation) | toy MoE Baseline · toy models, m=16 | SRP: 43.56 -> 41.62 | Hurt |
| make the first layer dense or interleave dense layers (ablation) | toy MoE Baseline · toy models, m=16 | SRP: 43.56 -> 44.87 (Skip1), 43.67 (Sparse2) | No clear effect |
| increase cache size from 1x to 2x active experts | oracle segment cache (SCH) · Baseline toy model, m=16 | relative hit rate: 80.97 -> 90.55 (3x: 96.23) | Improved |
| use oracle segment cache instead of LRU or LFU | SCH vs LRU/LFU · Baseline toy model, 2x cache | relative hit rate: 67.04 (LRU), 70.87 (LFU) -> 90.55 (SCH) | Improved |
| apply SFT, DPO or instruction tuning after pretraining (ablation) | OLMoE · REAL models, SRP | SRP: +0.47 SRP at m=16 after SFT | No clear effect |
| domain-specialized experts vs vocabulary-specialized experts | MoE LLMs · REAL models | SRP correlation | Improved |
Closest papers by meaning
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.93)
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.90)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.90)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.90)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
