Research map / Architectures and efficiency / RNNs and normalization
SAPipe: Staleness-Aware Pipeline for Data Parallel DNN Training
Yangrui Chen, Cong Xie, Meng Ma, Juncheng Gu, Yanghua Peng, Haibin Lin, Chuan Wu, Yibo Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add staleness compensation (SAPipe) over 1-step stale PipeSGD | PipeSGD · Transformer Multi30K | accuracy: 0.531 -> 0.668 | Improved |
| add staleness compensation over PipeSGD | PipeSGD · ResNet50 CIFAR-10 | accuracy: 0.893 -> 0.932 | Improved |
| add staleness compensation over PipeSGD | PipeSGD · VGG16 CIFAR-10 | accuracy: 0.906 -> 0.926 | Improved |
| add WP-OPT3 over PipeSGD | PipeSGD · GPT-2 WikiText-2 | perplexity: 22.36 -> 20.23 | Improved |
| use delay compensation only (SAPipe-DC) instead of PipeSGD | PipeSGD · Transformer Multi30K | accuracy: 0.531 -> 0.526 | No clear effect |
| WP-OPT2 instead of PipeSGD | PipeSGD · GPT-2 WikiText-2 | perplexity: 22.36 -> 22.60 | Hurt |
| WP-OPT1 vs OPT3 option choice | SAPipe · ImageNet VGG16 | accuracy: 0.718 -> 0.735 | Mixed |
| SAPipe-DC versus non-stale BytePS | SAPipe-DC · ResNet50 CIFAR-10 | accuracy: 0.932 -> 0.894 | Hurt |
| SAPipe-WP-OPT3 versus BytePS | SAPipe-WP-OPT3 · VGG16 ImageNet | accuracy: 0.731 -> 0.735 | Improved |
| SAPipe throughput versus BytePS | BytePS · throughput, 8-64 GPUs | speedup: up to 57% | Improved |
| SAPipe throughput versus Horovod | Horovod · throughput, 8-64 GPUs | speedup: up to 203% | Improved |
| remove runtime optimization (kernel fusion, buffers) (ablation) | SAPipe · throughput, 32 GPUs | training speed: decreases the training speed by 7% to 24% | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- PipeSGD — Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (NeurIPS 2018) · better on 6 of 6 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- On the Acceleration of Deep Learning Model Parallelism With Staleness (CVPR 2020 · similarity 0.90)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.87)
- Practical Performance Guarantees for Pipelined DNN Inference (ICML 2024 · similarity 0.85)
- Pfeife: Automatic Pipeline Parallelism for PyTorch (ICML 2025 · similarity 0.85)
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.85)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
