AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Taming Sparsely Activated Transformer with Stochastic Experts

Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, Tuo Zhao

ICLR 2022 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 27 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use randomly selected experts with a consistency-regularized training objective instead of conventional Transformer or Switch routingTHOR · 12 low-resource translation tasksBLEU: more than 1.0 BLEU score improvement ... in all the 12 tasksImproved
Use THOR rather than a same-sized Switch modelMultilingual translation · 20 translation tasksAverage BLEU: 22.4 -> 24.4Improved
Scale Switch to 32 experts and 5.5B parameters instead of THOR with 4 experts and 300M parametersMultilingual translation · 20 translation tasksAverage BLEU: 24.4 -> 24.4No clear effect
Add the consistency regularizer (ablation)THOR training objective · Translation ablation experimentsBLEUImproved
Remove one of the two cross-entropy terms (ablation)THOR training objective · Translation ablation experimentsBLEU: very small BLEU score lossHurt
Use sentence-level rather than token-level random routing at inference (ablation)THOR · Translation inference methodsBLEUNo clear effect
Ensemble predictions across all experts instead of dispatching to one expert (ablation)THOR · Translation inference methodsBLEUImproved
Increase consistency-regularization strength above 2.0 (ablation)THOR · Cs-En low-resource translationBLEUNo clear effect
Use consistency regularization during training to reduce randomness in predictions (ablation)THOR versus Switch Transformer with router removed at inference · 20-run inference comparisonprediction variance: 0.008; THOR below 0.002Improved
Remove the trained router at inference (ablation)Switch Transformer · Cs-En translationBLEU: 20.6 -> 20.4Hurt
Increase the number of THOR experts from 2 to 8 (ablation)THOR · De-En low-resource translationBLEUImproved
Increase Transformer FFN hidden dimension from 2k to 8k (ablation)Transformer · De-En low-resource translationBLEUHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy