Research map / Optimization / Distributed and compressed
From Promise to Practice: Realizing High-performance Decentralized Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes when decentralized training beats All-Reduce in wall-clock time via a runtime model, and proposes DAdam (decentralized Adam with overlapped communication/computation) plus AccumAdam (gradient accumulation to reduce variance), validated on transformer, ResNet-50, and GPT-2 training on up to 64 GPUs.
Problem
Decentralized training is theoretically more scalable than All-Reduce, but its practical speedups are poorly understood, its generalization often lags All-Reduce, and no well-suited decentralized Adam exists for training transformers.
Approach
Identify three factors (overlap of communication with computation via bucketed, neighbor-independent updates; heterogeneous communication costs with an alternating exponential ring topology; resilience to computation-time variance by avoiding intra-node gradient sync) and build a runtime model; pair this with a decentralized Adam (DAdam) whose search direction depends only on local state, plus an
Key results
- On WMT14 En-De with transformer (base), AccumAdam with the Complete topology reached 27.80 BLEU vs 27.79 for All-Reduce, and DAdam with AER reached 27.58 BLEU.
- On ImageNet-1K with ResNet-50 (32 workers), AccumAdam with AER reached 76.54 top-1 vs 76.01 for All-Reduce.
- On GPT-2 (small) pretraining with 4x4 A40 GPUs, decentralized AccumAdam achieved 30.41 hours vs 32.46 hours for All-Reduce (about 6.31% relative speedup) at similar validation loss (2.8561 vs 2.8626).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add gradient accumulation (AccumAdam) instead of DAdam (ablation) | Decentralized Adam on WMT14 En-De, Transformer base · WMT14 En-De test | METEOR: 0.5440 ± 0.0013 -> 0.5481 ± 0.0027 | Helped |
| Add gradient accumulation (AccumAdam) instead of DAdam (ablation) | Decentralized Adam on WMT14 En-Fr, Transformer big · WMT14 En-Fr test | BLEU: 45.60 ± 0.12 -> 46.05 ± 0.03 | Helped |
| Use AccumAdam instead of DAdam (ablation) | Decentralized training on ImageNet-1K · ImageNet-1K validation | Top-1 accuracy: 75.31 -> 76.54 | Helped |
| Use DAdam instead of AllReduce | Transformer base on WMT14 En-De · WMT14 En-De test | BLEU: 27.79 ± 0.16 -> 27.58 ± 0.06 | No clear effect |
| Use AccumAdam instead of AllReduce | Transformer big on WMT14 En-Fr · WMT14 En-Fr test | BLEU: 45.70 ± 0.30 -> 46.05 ± 0.03 | Helped |
| Use AccumAdam instead of AllReduce | ResNet-50 decentralized training · ImageNet-1K validation | Top-1 accuracy: 76.01 -> 76.54 | Helped |
| Use AccumAdam with AER instead of AllReduce | GPT-2 small pretraining · OpenWebText, 600k steps | validation loss: 2.8626 vs. 2.8561 | Helped |
| Use decentralized AccumAdam with AER instead of AllReduce | GPT-2 small pretraining · OpenWebText, 600k steps | runtime: 32.461 hours vs. 30.41 hours | Helped |
| Use Alternating Exponential Ring instead of Complete topology (ablation) | Decentralized model averaging, 100Gbps setup · 16 GPUs, 8196 averaging iterations | communication time: 55.31 s -> 29.55 s | Helped |
| Remove dependence of the local update on neighbors' same-iteration gradients (ablation) | Decentralized training · Transformer, 2×4 A100, 100K-token global batch | per-iteration runtime: 225.09 ms -> 200.43 ms; 10.95% | Helped |
| Use AccumAdam with AER instead of AllReduce | GPT-2 small pretraining · OpenWebText, 600k steps | training loss: 2.8468 vs. 2.8419 | Helped |
Surprising findings
- Decentralized training with a sparse topology showed speedups mainly when communication-bound; in the A100 100Gbps setup the one-peer exponential topology was slower than the complete topology for averaging.
- Removing the intra-node gradient synchronization dependency reduced per-iteration runtime by 10.95% in a transformer setup with fast networks.
Limitations
- Speedup on GPT-2 is modest (about 6%) because workloads were balanced; larger gains are speculated only for imbalanced batching.
- The PyTorch extension supports only one GPU per worker and lacks automatic mixed precision and sparse gradient support.
- Theory bound includes an extra term that depends on the step size and is only negligible for sufficiently small step sizes; topology choice is tuned to specific cluster network characteristics.
Experimental setup
Datasets: WMT14 English-German WMT14 English-French ImageNet-1K OpenWebText
Benchmarks: WMT14 newstest2014 ImageNet-1K validation OpenWebText validation loss
Metrics: BLEU METEOR Top-1 accuracy Top-5 accuracy training loss validation loss per-iteration runtime wall-clock speedup
Baselines: All-Reduce Adam (DDP) Stochastic Gradient Push (1-OSGP) DAdam Complete topology
Builds on: Adam PyTorch DistributedDataParallel Stochastic Gradient Push Decentralized SGD
Closest papers by meaning
- Consensus Control for Decentralized Deep Learning (ICML 2021 · similarity 0.92)
- A Decentralized Parallel Algorithm for Training Generative Adversarial Nets (NeurIPS 2020 · similarity 0.91)
- DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models (CVPR 2024 · similarity 0.88)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.87)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.87)
- Decentralized Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.