Research map / Architectures and efficiency / Mixture of experts
Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
Tianlong Chen, Zhenyu Zhang, AJAY KUMAR JAISWAL, Shiwei Liu, Zhangyang Wang
This paper tried a common change: Add dropout
Use SMoE-Dropout full capacity instead of dense dropout training · ASDiv-A · problem solving rate: 55.27 -> 56.30 Improved
What 795 accepted papers report for this change
- 549 improved
- 87 no clear effect
- 12 mixed
- 147 hurt
- Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference ICML 2026 · apply layer dropout to whole transformer blocks instead of attention and FFN sub-blocks separately · 20 TPP val loss, 271M · val loss: 2.377 -> 2.367Improved
- Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles ICML 2026 · use MC Dropout · vision and SST-2 · accuracy: 86.3 -> 76.6 on Flowers102Hurt
- Motion Planning in Compressed Representation Spaces ICML 2026 · add nested dropout during training · WOMD guided generation · goal FDE: 4.145 -> 1.104Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with SMoE-Dropout rather than dense dropout training | Transformer-XL · enwik8 | BPC: 1.1623 -> 1.1486 | Improved |
| Train with SMoE-Dropout rather than dense dropout training | BERT · BookCorpus | BPC: 7.6546 -> 7.6293 | Improved |
| Train with SMoE-Dropout rather than dense dropout training | RoBERTa · BookCorpus | BPC: 8.0903 -> 6.5491 | Improved |
| Use SMoE-Dropout full capacity instead of dense dropout training | BERT · ASDiv-A | problem solving rate: 55.27 -> 56.30 | Improved |
| Use SMoE-Dropout full capacity instead of dense dropout training | BERT · SVAMP | problem solving rate: 34.24 -> 35.33 | Improved |
| Use SMoE-Dropout full capacity instead of dense dropout training | RoBERTa · SVAMP | problem solving rate: 28.90 -> 33.94 | Improved |
| Add gradual increase of activated experts during training (ablation) | SMoE · SST-2 | accuracy: 80.06 -> 79.40; 80.79 -> 81.02; 80.58 -> 81.13; 80.99 -> 81.71; 81.20 -> 81.51 | Improved |
| Replace the learnable router with a fixed randomly initialized router (ablation) | SMoE with gradually increased k · SST-2 | accuracy: 79.40 -> 79.02; 81.02 -> 81.25; 81.13 -> 82.00; 81.71 -> 82.03; 81.51 -> 82.32 | Mixed |
| Remove gradual increase of activated experts (ablation) | SMoE-Dropout · enwik8 | BPC | Hurt |
| Use random initialized fixed-weight routing instead of fully random assignment or random hash routing (ablation) | Transformer-XL · enwik8 | BPC: 2.96~170.11 (× 10^-2) lower BPC | Improved |
| Place SMoE layers in later rather than earlier Transformer-XL layers (ablation) | SMoE-Dropout · enwik8 | BPC | Improved |
| Distill the trained model to a single expert (ablation) | SMoE-Dropout Transformer-XL · SST-2 | accuracy: 82.01 | Improved |
Closest papers by meaning
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.89)
- From Sparse to Soft Mixtures of Experts (ICLR 2024 · similarity 0.88)
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR 2024 · similarity 0.88)
- Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training (NeurIPS 2025 · similarity 0.87)
- Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models (ICLR 2025 · similarity 0.87)
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
