Research map / Architectures and efficiency / Mixture of experts
MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar, Samuel Horváth
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MoE with MoSE at full width (w=1.0) | GPT2-Small 3B tokens · OpenWebText | perplexity: 41.82 -> 39.02 | Improved |
| replace MoE with MoSE at full width (w=1.0) | GPT2-Small 15B tokens · OpenWebText | perplexity: 31.03 -> 31.26 | No clear effect |
| add test-time width identification (TTT) (ablation) | MoSE GPT2-Standard 15B · OpenWebText | perplexity: 20.65 -> 20.38 | Improved |
| add test-time width identification (TTT) (ablation) | MoSE GPT2-Standard 15B · LAMBADA | accuracy: 0.336 -> 0.338 | Improved |
| use MoSE (TTT) instead of MoE | GPT2-Medium · HellaSwag | accuracy: 0.335 -> 0.385 | Improved |
| use MoSE (TTT) instead of MoE | GPT2-Medium · PIQA | accuracy: 0.675 -> 0.710 | Improved |
| use MoSE (TTT) instead of MoE | GPT2-Medium · SIQA | accuracy: 0.290 -> 0.305 | Improved |
| router-conditioned normalized-probability widths instead of uniform widths (ablation) | MoSE inference · OpenWebText Pareto frontier | perplexity at fixed FLOPs | Improved |
| learn sharpness gamma by test-time training instead of fixed gamma=1.0 (ablation) | MoSE inference · OpenWebText Pareto frontier | perplexity at fixed FLOPs | Improved |
| layer-wise gamma instead of shared gamma (ablation) | MoSE TTT · OpenWebText Pareto frontier | perplexity | Improved |
| scale to GPT2-Large 2.5B with MoSE | MoSE · OpenWebText | FLOPs at matched perplexity: 32.8% lower FLOPs | Improved |
| continual pretraining slimmability adaptation | DeepSeek-V2-Lite 16B · validation perplexity | FLOPs at comparable perplexity: 38.1% lower FLOPs | Improved |
Closest papers by meaning
- Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.90)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- OLMoE: Open Mixture-of-Experts Language Models (ICLR 2025 · similarity 0.89)
- Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025 · similarity 0.89)
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
