AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training

Chang Chen, Min Li, Zhihua Wu, Dianhai Yu, Chao Yang

NeurIPS 2022 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
uneven dispatch (more data to neighbor device) instead of even dispatchMoE all-to-all · [2,2] symmetric tree, 128MBcommunication time (us): 14019 -> 10765Improved
topology-aware loss instead of load-balance lossFastMoE · openwebtext2 8 expertsvalid PPL: 18.12 -> 17.97Improved
topology-aware loss instead of load-balance lossFastMoE · openwebtext2 16 expertsvalid PPL: 15.39 -> 15.18Improved
topology-aware loss instead of load-balance lossFastMoE · openwebtext2 32 expertsvalid PPL: 13.53 -> 13.37Improved
topology-aware loss instead of load-balance lossFastMoE · openwebtext2 48 expertsvalid PPL: 12.49 -> 12.55Hurt
add TA-MoE to DeepSpeed-MoEDeepSpeed-MoE · GPT MoE across clusters A/B/Cthroughput speedup: 1.05x-1.61xImproved
add TA-MoE to FastMoEFastMoE · GPT MoE across clusters A/B/Cthroughput speedup: 1.01x-4.77xImproved
TA-MoE on cluster C with cross-switch contentionFastMoE · cluster Cthroughput speedup: 4.77xImproved
TA-MoE adaptive loss instead of FasterMoE Hir compulsory dispatchFasterMoE · cluster C, val loss 3.1time to convergence: 1.25xImproved
TA-MoE adaptive loss instead of FasterMoE Hir compulsory dispatchFasterMoE · cluster C, val loss 2.9time to convergence: 1.47xImproved
TA-MoE adaptive loss instead of FasterMoE Hir compulsory dispatchFasterMoE · cluster C, val loss 2.8time to convergence: 1.54xImproved
TA-MoE on communication costFastMoE · cluster Ccommunication cost speedup: 1.16x to 6.4xImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy