Research map / Architectures and efficiency / Mixture of experts
Rethinking Sparse Mixture of Experts from a Unified Perspective
Giang Do, Hung Le, Truyen Tran
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace token-choice routing with USMoE | Qwen3-30B-A3B-Thinking · six reasoning benchmarks, clean | avg accuracy: 0.706 -> 0.721 | Improved |
| replace token-choice routing with USMoE | Qwen3-30B-A3B-Instruct · six reasoning benchmarks, clean | avg accuracy: 0.724 -> 0.732 | Improved |
| replace token-choice routing with USMoE under 15% noise | Qwen3-30B-A3B-Thinking · six reasoning benchmarks, corrupt | avg accuracy: 0.408 -> 0.532 | Improved |
| replace token-choice routing with USMoE under 15% noise | Qwen3-30B-A3B-Instruct · six reasoning benchmarks, corrupt | avg accuracy: 0.421 -> 0.556 | Improved |
| replace token-choice with USMoE in OLMoE | OLMoE-1B-7B · MTEB subset | avg score: 48.0 -> 53.2 | Improved |
| replace token-choice with USMoE in DeepSeekMoE | DeepSeekMoE-16B · MTEB subset | avg score: 44.0 -> 52.0 | Improved |
| replace Qwen1.5-MoE token-choice with USMoE | Qwen1.5-MoE-A2.7B · MTEB subset | avg score: 45.4 -> 53.4 | Improved |
| use USMoE with fractional k=1.5 instead of TC with k=2 | Transformer-XL 20M · Text8 and WikiText-103 | BPC / perplexity: Text8 1.29 -> 1.28; WikiText-103 30.16 -> 30.67 | Mixed |
| use USMoE instead of TC and EC in ViT-MoE | ViT-MoE · five image datasets avg accuracy | accuracy: 74.4 -> 77.3 | Improved |
| add Unified Score on top of Unified Mechanism (ablation) | USMoE · six reasoning benchmarks, clean | avg accuracy: 0.730 -> 0.732 | Improved |
| add Unified Mechanism (global allocation) to TC (ablation) | Qwen3-30B-A3B-Instruct · six reasoning benchmarks, clean | avg accuracy: 0.724 -> 0.730 | Improved |
| use sequence-level instead of batch-level unified mechanism (ablation) | USMoE · Tweet classification | accuracy: 51.7 -> 52.5 | Improved |
Closest papers by meaning
- Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025 · similarity 0.90)
- Multi-Head Mixture-of-Experts (NeurIPS 2024 · similarity 0.90)
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR 2024 · similarity 0.90)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.89)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
- MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
