AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers

Tianlong Chen, Zhenyu Zhang, AJAY KUMAR JAISWAL, Shiwei Liu, Zhangyang Wang

ICLR 2023 · Architectures and efficiency › Mixture of experts · approach: Sparse MoE as regularization and attention · 6 citations (OpenAlex)

This paper tried a common change: Add dropout

Use SMoE-Dropout full capacity instead of dense dropout training · ASDiv-A · problem solving rate: 55.27 -> 56.30 Improved

What 795 accepted papers report for this change

See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Train with SMoE-Dropout rather than dense dropout trainingTransformer-XL · enwik8BPC: 1.1623 -> 1.1486Improved
Train with SMoE-Dropout rather than dense dropout trainingBERT · BookCorpusBPC: 7.6546 -> 7.6293Improved
Train with SMoE-Dropout rather than dense dropout trainingRoBERTa · BookCorpusBPC: 8.0903 -> 6.5491Improved
Use SMoE-Dropout full capacity instead of dense dropout trainingBERT · ASDiv-Aproblem solving rate: 55.27 -> 56.30Improved
Use SMoE-Dropout full capacity instead of dense dropout trainingBERT · SVAMPproblem solving rate: 34.24 -> 35.33Improved
Use SMoE-Dropout full capacity instead of dense dropout trainingRoBERTa · SVAMPproblem solving rate: 28.90 -> 33.94Improved
Add gradual increase of activated experts during training (ablation)SMoE · SST-2accuracy: 80.06 -> 79.40; 80.79 -> 81.02; 80.58 -> 81.13; 80.99 -> 81.71; 81.20 -> 81.51Improved
Replace the learnable router with a fixed randomly initialized router (ablation)SMoE with gradually increased k · SST-2accuracy: 79.40 -> 79.02; 81.02 -> 81.25; 81.13 -> 82.00; 81.71 -> 82.03; 81.51 -> 82.32Mixed
Remove gradual increase of activated experts (ablation)SMoE-Dropout · enwik8BPCHurt
Use random initialized fixed-weight routing instead of fully random assignment or random hash routing (ablation)Transformer-XL · enwik8BPC: 2.96~170.11 (× 10^-2) lower BPCImproved
Place SMoE layers in later rather than earlier Transformer-XL layers (ablation)SMoE-Dropout · enwik8BPCImproved
Distill the trained model to a single expert (ablation)SMoE-Dropout Transformer-XL · SST-2accuracy: 82.01Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy