Research map / Architectures and efficiency / RNNs and normalization
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, Alexander Schwing
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use decentralized pipelined SGD with one-iteration-delayed updates instead of D-Sync | Distributed deep-net training · Four benchmarks | wall-clock speedup: 2.0~3.2× | Improved |
| Use decentralized pipelined SGD with one-iteration-delayed updates instead of PS-Sync | Distributed deep-net training · Four benchmarks | wall-clock speedup: 4.0~5.4× | Improved |
| Replace PS-Sync with D-Sync | Distributed training · CIFAR100-Convex | convergence speed: 40% | Improved |
| Replace D-Sync with Pipe-SGD | Distributed training · CIFAR100-Convex | convergence speed: 37% | Improved |
| Add pipelining to D-Sync with truncation | CIFAR100-Convex · CIFAR100-Convex | convergence speed: 46% | Improved |
| Use decentralized balanced communication instead of a centralized parameter server | Distributed training · Four-node GPU cluster | uncompressed communication time: 50% | Improved |
| Add 16-bit truncation to Pipe-SGD | Pipe-SGD · AlexNet | top-1 accuracy: -0.005 | Hurt |
| Add lossy compression to Pipe-SGD | Pipe-SGD · Benchmarks other than AlexNet | top-1 accuracy | Improved |
| Increase iteration dependency beyond K = 2 (ablation) | Pipe-SGD · Distributed training | wall-clock training time | Hurt |
| Use sequential rather than pipelined gradient communication (ablation) | Pipe-SGD when communication-bound · Distributed training | wall-clock training time | Improved |
| Use complex compression instead of lightweight truncation or quantization (ablation) | AllReduce · Pipe-SGD compression benchmarks | compression overhead relative to uncompressed communication: 1.6~2.3× the uncompressed communication time; 25.6~36.8× the compressed communication time | Hurt |
| Use warm-up before enabling pipelining (ablation) | Pipe-SGD on AlexNet and ResNet · Large non-convex models | final accuracy | Improved |
Later papers that compared with this method
1 later accepted paper reports PipeSGD as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- SAPipe: Staleness-Aware Pipeline for Data Parallel DNN Training (NeurIPS 2022 · better on 6 of 6 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices (NeurIPS 2021 · similarity 0.88)
- DropCompute: simple and more robust distributed synchronous training via compute variance reduction (NeurIPS 2023 · similarity 0.87)
- FRESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training (NeurIPS 2026 · similarity 0.87)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.86)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.85)
- FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
