Research map / Optimization / Distributed and compressed
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace centralized parameter-server communication with neighbor communication | PSGD · Theoretical complexity | communication complexity on the busiest node: O(n) -> O (Deg(network)) | Helped |
| Use decentralized neighbor averaging rather than centralized mini-batch aggregation | PSGD · Theoretical complexity | computational complexity: O( n/ϵ + 1/ϵ² ) -> O( n/ϵ + 1/ϵ² ) | No clear effect |
| Use D-PSGD instead of centralized SGD on a low-bandwidth or high-latency network | ResNet-20 training · CIFAR-10 | wall-clock speed: up to 10× faster | Helped |
| Scale D-PSGD from one GPU to multiple GPUs | D-PSGD · ResNet-20, 7 GPUs | speedup: up to 5× speed up | Helped |
Closest papers by meaning
- Asynchronous Decentralized Parallel Stochastic Gradient Descent (ICML 2018 · similarity 0.92)
- Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent (AAAI 2024 · similarity 0.89)
- Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithm (ICML 2024 · similarity 0.88)
- Enhancing Parallelism in Decentralized Stochastic Convex Optimization (ICML 2025 · similarity 0.87)
- Implicit Regularization of Decentralized Gradient Descent for Sparse Regression (NeurIPS 2024 · similarity 0.87)
- Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers for Faster Convergence (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.