Research map / Optimization / Distributed and compressed
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| average worker models every I iterations, with I ≤ T^{1/4}/N^{3/4}, rather than synchronize every iteration | PR-SGD · Non-convex stochastic optimization | communication rounds: O(T^{1/4}/N^{3/4}) | Helped |
| use any fixed constant I (ablation) | PR-SGD · Non-convex stochastic optimization | average expected squared gradient norm: O(1/√(NT) + N/T) | Mixed |
| choose I ≤ T^{1/4}/N^{3/4} (ablation) | PR-SGD · Non-convex stochastic optimization | average expected squared gradient norm: O(1/√(NT)) | Helped |
| use time-varying epoch lengths and learning rates K^s = ⌈s^{1/3}/N⌉ and γ^s = N/s^{2/3} (ablation) | PR-SGD · Non-convex stochastic optimization | weighted average expected squared gradient norm: O(log(NT)/√(NT)) | Helped |
| allow faster workers to take more local SGD steps per epoch | PR-SGD in heterogeneous networks · Heterogeneous networks with identical worker data distributions | weighted average expected squared gradient norm: O(1/√(NT)) | Helped |
| replace worker-specific epoch lengths with the slowest worker's common epoch length (ablation) | PR-SGD in heterogeneous networks · Heterogeneous networks | early-epoch convergence | Hurt |
| periodically average models with I ∈ {4, 8, 16, 32} rather than synchronize every iteration | ResNet20 · CIFAR10 | training loss | Mixed |
Closest papers by meaning
- Communication-efficient SGD: From Local SGD to One-Shot Averaging (NeurIPS 2021 · similarity 0.91)
- Local SGD Converges Fast and Communicates Little (ICLR 2019 · similarity 0.90)
- Don't Use Large Mini-batches, Use Local SGD (ICLR 2020 · similarity 0.90)
- LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging (NeurIPS 2026 · similarity 0.87)
- A Quadratic Synchronization Rule for Distributed Deep Learning (ICLR 2024 · similarity 0.86)
- Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.