Research map / Architectures and efficiency / Mixture of experts
On the Representation Collapse of Sparse Mixture of Experts
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, Furu Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace dot-product routing with low-dim L2-normalized routing and learnable temperature (X-MoE) | SMoE (softmax gating) · XTREME average | avg score: 63.8 -> 65.3 | Improved |
| replace dot-product routing with X-MoE routing | SMoE (sigmoid gating) · XTREME average | avg score: 64.1 -> 65.0 | Improved |
| use X-MoE routing | SMoE · WMT-10 x->en | avg BLEU: 29.7 -> 30.2 | Improved |
| add sparse MoE layer to dense model | Dense Transformer · XTREME average | avg score: 61.4 -> 63.8 | Improved |
| X-MoE routing vs baseline on upstream MLM | SMoE · mC4 MLM validation | perplexity | Improved |
| add dimension reduction, L2 norm and frozen routing (each component) (ablation) | X-MoE sigmoid · XNLI and MLQA | fine-tuning score | Improved |
| routing dimension N/2 vs other sizes (N/4..4N) (ablation) | X-MoE sigmoid · XNLI / MLQA | score | Mixed |
| add load balancing loss during fine-tuning (ablation) | X-MoE · XNLI and MLQA | score | No clear effect |
Closest papers by meaning
- Tight Clusters Make Specialized Experts (ICLR 2025 · similarity 0.89)
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.88)
- Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025 · similarity 0.88)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
