Research map / Architectures and efficiency / RNNs and normalization
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient
Max Ryabinin, Tim Dettmers, Michael Diskin, Alexander Borzunov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase model size (d_model 768 to 12288) at 500 Mb/s | pipeline-parallel Transformer layers · V100 pipeline, no latency | relative GPU utilization: 18.0% to 82.1% | Improved |
| add 8-bit activation compression with 3 layers per stage (ablation) | xxlarge-sized Transformer (d_model=4096) · V100 pipeline, no latency, 500 Mb/s | relative GPU utilization: 32.1% to 89.5% | Improved |
| increase network latency to 100ms RTT | large pipeline models · V100 pipeline, 500 Mb/s | relative GPU utilization: GPT-3 82.1% to 60.2%; Ours 89.5% to 71.5% | Hurt |
| SWARM (stochastic wiring) vs GPipe, GPT-3 layers with 100ms latency | SWARM parallelism · GPT-3 (4 layers), 100ms latency | training throughput min/batch (lower is better): 186.7 vs 218.4 (GPipe) | Improved |
| SWARM vs GPipe on xxlarge layers, 100ms latency | SWARM parallelism · xxlarge (4 layers), 100ms latency | training throughput min/batch (lower is better): 48.2 vs 108.8 (GPipe) | Improved |
| SWARM vs ZeRO-Offload on small models (xxlarge, GPT-3 layers) | SWARM parallelism · xxlarge and GPT-3 (4 layers), no latency | training throughput min/batch (lower is better): xxlarge 44.2 vs Offload 33.8; GPT-3 168.3 vs Offload 272.7 (SWARM faster on GPT-3) | Hurt |
| adaptive rebalancing with period T=300 s vs no rebalancing (ablation) | SWARM pipeline on preemptible T4 peers · 32-hour simulation from preemptible T4 data | % of optimal throughput (overall): 82.7 to 95.8 | Improved |
| reduce rebalancing period from 300 s to 60 s (ablation) | SWARM adaptive rebalancing · 32-hour simulation from preemptible T4 data | % of optimal throughput (overall): 95.8 to 97.6 | Improved |
| 8-bit blockwise activation/gradient compression plus layer sharing for 1B model | SWARM on 400 preemptible T4 nodes · Pipeline throughput, layer sharing | samples/s: 17.6 actual vs 19.2 best-case | No clear effect |
| remove layer sharing (default Transformer blocks) with same 8-bit compression (ablation) | SWARM on T4 nodes · Pipeline throughput | samples/s: 17.6 (layer sharing) vs 8.8 (default Transformer) | Hurt |
| SWARM with 1B shared model trained on preemptible T4s versus data-parallel offloading on 128 A100s | SWARM parallelism · Pile, learning curves over ~4 weeks | training dynamics: similar learning curves | No clear effect |
| scale number of T4 nodes from 8 to 128 | SWARM parallelism · Scaling evaluation, T4 only | pipeline throughput: approximately linear scaling | Improved |
Closest papers by meaning
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.88)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.88)
- Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping (ICML 2025 · similarity 0.87)
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.87)
- CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · similarity 0.86)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
