Research map / Optimization / Distributed and compressed
A Quadratic Synchronization Rule for Distributed Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Quadratic Synchronization Rule (QSR), which sets the local-step period H proportional to 1/eta^2 as the learning rate decays, so local gradient methods both cut communication and improve test accuracy on ImageNet.
Problem
Choosing the synchronization period H in local gradient methods (e.g., Local SGD/AdamW) to reduce communication while improving, rather than hurting, generalization in distributed data-parallel deep learning.
Approach
Dynamically set H_s = max{H_base, floor((alpha/eta_t)^2)} so the sync period grows quadratically as the learning rate decays; the power 2 is motivated by a slow-SDE analysis showing H ~ eta^-2 yields a K-times-larger sharpness-reducing drift than SGD or H ~ eta^-1.
Key results
- On ViT-B with batch 4096 on 2x8 GPUs, Local AdamW + QSR reaches 80.98% vs 79.86% for parallel AdamW while using 10.4% of communication volume.
- QSR cuts ViT-B training time from 26.7 to 20.2 hours on 2x8 GPUs and from 8.6 to 5.5 hours on 8x8 GPUs.
- With ResNet-152, Local SGD + QSR reaches 80.30% vs 79.53% for parallel SGD at 39.7% communication (H=2 base).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QSR instead of constant H=2 | Local SGD on ResNet-152 · ImageNet, batch size 4096, cosine | validation accuracy (%): 79.54 (0.07) -> 80.30 (0.04) | Helped |
| Use QSR instead of constant H=4 | Local AdamW on ViT-B · ImageNet, batch size 4096, cosine | validation accuracy (%): 79.32 (0.06) -> 80.98 (0.05) | Helped |
| Use QSR instead of parallel training | AdamW on ViT-B · ImageNet, batch size 4096, cosine | validation accuracy (%): 79.86 (0.03) -> 80.98 (0.05) | Helped |
| Use QSR instead of constant H=2 | Local SGD on ResNet-152 · ImageNet, batch size 16384, cosine | validation accuracy (%): 78.67 -> 79.27 | Helped |
| Use QSR instead of constant H=4 | Local AdamW on ViT-B · ImageNet, batch size 16384, cosine | validation accuracy (%): 77.83 -> 79.36 | Helped |
| Use QSR instead of constant H=2 | Local SGD on ResNet-152 · ImageNet, batch size 4096, step decay | validation accuracy (%): 79.58 -> 80.40 | Helped |
| Use QSR instead of parallel training | AdamW on ViT-B · ImageNet, batch size 4096, step decay | validation accuracy (%): 79.91 -> 80.9 | Helped |
| Use QSR instead of parallel training | SGD on ResNet-152 · ImageNet, 200 epochs, 2x8 GPUs | total training time (h): 20.7 -> 18.0 | Helped |
| Use QSR instead of parallel training | AdamW on ViT-B · ImageNet, 300 epochs, 8x8 GPUs | total training time (h): 8.6 -> 5.5 | Helped |
| Use QSR instead of constant H=2 | Local SGD on ResNet-50 · ImageNet, 90 epochs, cosine | validation accuracy (%): 76.60 -> 76.65 | No clear effect |
| Increase the QSR minimum synchronization period from H_min=2 to H_min=4 (ablation) | QSR Local SGD on ResNet-152 · ImageNet, batch size 16384, cosine | validation accuracy (%): 79.27 -> 78.65 | Hurt |
| Increase the QSR minimum synchronization period from H_min=4 to H_min=8 (ablation) | QSR Local AdamW on ViT-B · ImageNet, batch size 4096, cosine | validation accuracy (%): 80.98 (0.05) -> 80.56 (0.10) | Hurt |
Surprising findings
- QSR yields higher validation accuracy despite higher training loss, suggesting the gain comes from implicit regularization of local gradient noise rather than better optimization.
- QSR gives negligible gains for 90-epoch ResNet-50 training, where it matches but does not beat constant-H Local SGD.
Limitations
- Gains are limited for smaller models trained for shorter horizons (e.g., 90-epoch ResNet-50).
- Theory is based on a quasistatic SDE approximation with assumptions (smooth loss, manifold of minimizers) and does not directly cover language model pretraining.
- Effectiveness is shown only on ImageNet with ResNet-152 and ViT-B; the growth coefficient alpha and base H require per-setting tuning.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet
Metrics: top-1 validation accuracy communication volume wall-clock training time train loss
Baselines: Parallel SGD Parallel AdamW Local SGD with constant H Local AdamW with constant H Post-local SGD H ~ eta^-1 scaling cubic rule
Builds on: Local SGD Post-local SGD SDE approximation of SGD (slow SDE) Linear Scaling Rule
Closest papers by meaning
- $\textbf{A}^2\textbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning (NeurIPS 2023 · similarity 0.89)
- AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning (NeurIPS 2020 · similarity 0.89)
- Learning Efficient Parameter Server Synchronization Policies for Distributed SGD (ICLR 2020 · similarity 0.89)
- Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations (NeurIPS 2019 · similarity 0.88)
- Local SGD Converges Fast and Communicates Little (ICLR 2019 · similarity 0.88)
- LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.