AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient

Max Ryabinin, Tim Dettmers, Michael Diskin, Alexander Borzunov

ICML 2023 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 6 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase model size (d_model 768 to 12288) at 500 Mb/spipeline-parallel Transformer layers · V100 pipeline, no latencyrelative GPU utilization: 18.0% to 82.1%Improved
add 8-bit activation compression with 3 layers per stage (ablation)xxlarge-sized Transformer (d_model=4096) · V100 pipeline, no latency, 500 Mb/srelative GPU utilization: 32.1% to 89.5%Improved
increase network latency to 100ms RTTlarge pipeline models · V100 pipeline, 500 Mb/srelative GPU utilization: GPT-3 82.1% to 60.2%; Ours 89.5% to 71.5%Hurt
SWARM (stochastic wiring) vs GPipe, GPT-3 layers with 100ms latencySWARM parallelism · GPT-3 (4 layers), 100ms latencytraining throughput min/batch (lower is better): 186.7 vs 218.4 (GPipe)Improved
SWARM vs GPipe on xxlarge layers, 100ms latencySWARM parallelism · xxlarge (4 layers), 100ms latencytraining throughput min/batch (lower is better): 48.2 vs 108.8 (GPipe)Improved
SWARM vs ZeRO-Offload on small models (xxlarge, GPT-3 layers)SWARM parallelism · xxlarge and GPT-3 (4 layers), no latencytraining throughput min/batch (lower is better): xxlarge 44.2 vs Offload 33.8; GPT-3 168.3 vs Offload 272.7 (SWARM faster on GPT-3)Hurt
adaptive rebalancing with period T=300 s vs no rebalancing (ablation)SWARM pipeline on preemptible T4 peers · 32-hour simulation from preemptible T4 data% of optimal throughput (overall): 82.7 to 95.8Improved
reduce rebalancing period from 300 s to 60 s (ablation)SWARM adaptive rebalancing · 32-hour simulation from preemptible T4 data% of optimal throughput (overall): 95.8 to 97.6Improved
8-bit blockwise activation/gradient compression plus layer sharing for 1B modelSWARM on 400 preemptible T4 nodes · Pipeline throughput, layer sharingsamples/s: 17.6 actual vs 19.2 best-caseNo clear effect
remove layer sharing (default Transformer blocks) with same 8-bit compression (ablation)SWARM on T4 nodes · Pipeline throughputsamples/s: 17.6 (layer sharing) vs 8.8 (default Transformer)Hurt
SWARM with 1B shared model trained on preemptible T4s versus data-parallel offloading on 128 A100sSWARM parallelism · Pile, learning curves over ~4 weekstraining dynamics: similar learning curvesNo clear effect
scale number of T4 nodes from 8 to 128SWARM parallelism · Scaling evaluation, T4 onlypipeline throughput: approximately linear scalingImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy