Research map / Architectures and efficiency / Mixture of experts
Taming Sparsely Activated Transformer with Stochastic Experts
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, Tuo Zhao
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use randomly selected experts with a consistency-regularized training objective instead of conventional Transformer or Switch routing | THOR · 12 low-resource translation tasks | BLEU: more than 1.0 BLEU score improvement ... in all the 12 tasks | Improved |
| Use THOR rather than a same-sized Switch model | Multilingual translation · 20 translation tasks | Average BLEU: 22.4 -> 24.4 | Improved |
| Scale Switch to 32 experts and 5.5B parameters instead of THOR with 4 experts and 300M parameters | Multilingual translation · 20 translation tasks | Average BLEU: 24.4 -> 24.4 | No clear effect |
| Add the consistency regularizer (ablation) | THOR training objective · Translation ablation experiments | BLEU | Improved |
| Remove one of the two cross-entropy terms (ablation) | THOR training objective · Translation ablation experiments | BLEU: very small BLEU score loss | Hurt |
| Use sentence-level rather than token-level random routing at inference (ablation) | THOR · Translation inference methods | BLEU | No clear effect |
| Ensemble predictions across all experts instead of dispatching to one expert (ablation) | THOR · Translation inference methods | BLEU | Improved |
| Increase consistency-regularization strength above 2.0 (ablation) | THOR · Cs-En low-resource translation | BLEU | No clear effect |
| Use consistency regularization during training to reduce randomness in predictions (ablation) | THOR versus Switch Transformer with router removed at inference · 20-run inference comparison | prediction variance: 0.008; THOR below 0.002 | Improved |
| Remove the trained router at inference (ablation) | Switch Transformer · Cs-En translation | BLEU: 20.6 -> 20.4 | Hurt |
| Increase the number of THOR experts from 2 to 8 (ablation) | THOR · De-En low-resource translation | BLEU | Improved |
| Increase Transformer FFN hidden dimension from 2k to 8k (ablation) | Transformer · De-En low-resource translation | BLEU | Hurt |
Closest papers by meaning
- Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models (ICLR 2025 · similarity 0.87)
- Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models (ICLR 2025 · similarity 0.87)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.87)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.87)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.86)
- Monet: Mixture of Monosemantic Experts for Transformers (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
