Research map / Architectures and efficiency / Mixture of experts
MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
Xiaodong Chen, Mingming Ha, Zhenzhong Lan, Jing Zhang, Jianguo Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MoBE instead of MoLAE for compression | Qwen3-30B-A3B-2507 · 15-benchmark average | accuracy (%): 67.5 -> 75.8 | Improved |
| use MoBE instead of D2-MoE | Qwen3-30B-A3B-2507 · 15-benchmark average | accuracy (%): 66.4 -> 75.8 | Improved |
| use MoBE instead of MoLAE | Kimi-K2-Instruct · 15-benchmark average | accuracy (%): 76.6 -> 81.1 | Improved |
| use MoBE instead of MoLAE | DeepSeek-V3-0324 · 15-benchmark average | accuracy (%): 73.1 -> 78.0 | Improved |
| use MoBE instead of MoLAE | Qwen3-235B-A22B-2507 · 15-benchmark average | accuracy (%): 73.2 -> 80.9 | Improved |
| compress with MoBE vs uncompressed | Ling-Lite-Chat · 15-benchmark average | accuracy (%): 59.7 -> 58.6 | Hurt |
| reduce active experts (MoBE-dagger) vs MoBE (ablation) | Qwen3-235B-A22B-2507 · 15-benchmark average | accuracy (%): 80.9 -> 79.7 | Hurt |
| reduce active experts (MoBE-dagger) vs MoBE (ablation) | Qwen3-30B-A3B-2507 · 15-benchmark average | accuracy (%): 75.8 -> 75.7 | No clear effect |
| use ReLU activation instead of SiLU/Tanh in basis combination (ablation) | MoBE (Qwen3-30B gate matrices) · per-layer reconstruction MSE | MSE | Hurt |
| use Sigmoid instead of no activation (ablation) | MoBE (Qwen3-30B gate matrices) · per-layer reconstruction MSE | MSE | Hurt |
| add SiLU/Tanh/GELU activation vs none (ablation) | MoBE (Qwen3-30B gate matrices) · per-layer reconstruction MSE | MSE | Improved |
| add Z-score normalization of expert weights (ablation) | MoBE (Qwen3-30B gate matrices) · per-layer reconstruction MSE | MSE | Improved |
Later papers that compared with this method
2 later accepted papers report MoBE as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.
- Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density (ICML 2026 · better on 10 of 10 shared results)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · better on 4 of 4 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.93)
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.92)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.92)
- Delta Decompression for MoE-based LLMs Compression (ICML 2025 · similarity 0.91)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.91)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
