Research map / Optimization / Distributed and compressed
Scalable Distributed DL Training: Batching Communication and Computation
Shaoqi Wang, Aidi Pi, Xiaobo Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| batch parameter pulls and forward computation (iBatch) instead of default sequential PS | BigDL parameter server · VGG19-22K, 72 nodes | speedup: 26x -> 45x | Improved |
| batch layers (iBatch) instead of layer by layer overlap (ablation) | BigDL parameter server · VGG19-22K, 72 nodes | speedup: 32x -> 45x | Improved |
| layer by layer overlap instead of default sequential PS (ablation) | BigDL parameter server · GoogLeNet, 30 nodes | speedup: 26x -> 25x | Hurt |
| add Gradient Sparsification on top of iBatch | iBatch · Inception-V3 and VGG19, 72 nodes | speedup | Improved |
| increase cluster size to small (<=6 nodes) | iBatch vs baselines · all four networks | speedup | No clear effect |
Closest papers by meaning
- Sketching for Distributed Deep Learning: A Sharper Analysis (NeurIPS 2024 · similarity 0.87)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.86)
- Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis (NeurIPS 2025 · similarity 0.86)
- CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · similarity 0.86)
- A Quadratic Synchronization Rule for Distributed Deep Learning (ICLR 2024 · similarity 0.86)
- Communication Efficient Distributed Training with Distributed Lion (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
