Research map / Architectures and efficiency / Mixture of experts
Advancing Expert Specialization for Better MoE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds an orthogonality loss on top-k expert outputs and a routing-score variance loss to the standard MoE auxiliary loss, improving expert specialization and downstream accuracy during LoRA-based post-training on 11 benchmarks.
Problem
Standard auxiliary load-balancing losses in MoE push routing toward uniform expert usage, causing expert output overlap and routing uniformity that hinder expert specialization and degrade post-training performance.
Approach
Two complementary losses added to the auxiliary loss: an orthogonality loss penalizing projections between top-k expert outputs for the same token, and a negative-variance loss that maximizes the spread of routing scores per expert; the paper argues these are gradient-compatible with the auxiliary loss and with each other.
Key results
- Ours achieves the best accuracy on most tasks, e.g., GSM8K 35.00 vs 30.10 for ST-MoE on DeepSeek-MoE-16B and 87.62 vs 84.62 for GShard on Moonlight-16B-A3B.
- Claims up to 23.79% relative improvement over classic MoE baselines with auxiliary loss.
- Expert overlap reduced by up to 45% and routing score variance increased by over 150%, with load balance (MaxVio_global) matched to baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add orthogonality and variance losses | MoE post-training with auxiliary load balancing · Downstream benchmarks (11 benchmarks; three MoE models) | accuracy: 23.79% relative gains | Helped |
| Add orthogonality and variance losses | DeepSeek-MoE-16B · MMLU | accuracy: 34.23\std{2.20} -> 33.35\std{2.20} | Hurt |
| Add orthogonality and variance losses | MoE models trained with auxiliary load balancing · Load balancing across models | RMSE: 0.03 | No clear effect |
| Add orthogonality loss (ablation) | MoE post-training with auxiliary loss · Downstream benchmarks | accuracy | Helped |
| Add variance loss (ablation) | MoE post-training with auxiliary loss · Downstream benchmarks | accuracy | Helped |
| Use both orthogonality and variance losses rather than only one (ablation) | MoE post-training with auxiliary loss · Expert overlap across three models | expert overlap: below 0.3 | Helped |
| Add variance loss (ablation) | MoE post-training with auxiliary loss · Expert behavior across three models | expert orthogonality | Helped |
| Add orthogonality loss (ablation) | MoE post-training with auxiliary loss · Routing behavior across three models | routing score variance | Helped |
| Remove auxiliary loss (ablation) | MoE post-training · DeepSeek-V2-Lite load balancing | MaxVio_global: 2.48 -> 6.14 | Hurt |
| Remove auxiliary loss (ablation) | MoE post-training · Downstream benchmarks | accuracy | Mixed |
Surprising findings
- Removing the auxiliary loss entirely (w/o all) yields better downstream performance than keeping only the auxiliary loss, per the paper's ablation coverage analysis.
Limitations
- Evaluation limited to post-training with LoRA fine-tuning; pre-training and full-parameter settings not studied.
- Theoretical compatibility argued via gradient analysis and lemmas with informal assumptions.
- Authors note extension to vision, multimodal, and LoRA-MoE settings is unexplored.
Experimental setup
Datasets: Numina GLUE FLAN GSM8K MATH500 MMLU MMLU-pro BBH LiveBench GPQA HumanEval MBPP
Benchmarks: MMLU MMLU-pro BBH GLUE LiveBench GPQA HumanEval MBPP GSM8K MATH500 NuminaTest
Metrics: accuracy MaxVio_global silhouette coefficient expert overlap routing variance RMSE
Baselines: Aux Loss GShard ST-MoE Loss-Free Balancing
Builds on: auxiliary load balancing loss GShard ST-MoE Loss-Free Balancing
Closest papers by meaning
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.94)
- Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · similarity 0.92)
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.91)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.91)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.91)
- Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.