Research map / Optimization / Distributed and compressed
AdaGK-SGD: Adaptive Global Knowledge Guided Distributed Stochastic Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaGK-SGD, a distributed SGD method that extends global-knowledge correction to every local step via an adaptively set maximum lifetime (MLGK), with convergence analysis and CIFAR/ImageNet experiments.
Problem
Local methods reduce communication in distributed training but lose accuracy and generalization; global knowledge helps but is stale and costly to update every step, so it is typically used only at global averaging.
Approach
Defines a maximum lifetime (time-to-live) for global knowledge, adaptively set via cosine-similarity conditions or a training-loss-based rule, so a global-knowledge correction term guides every local update without extra communication; the MLGK module can be plugged into other local methods.
Key results
- On CIFAR-10 with ResNet, AdaGK-SGD reaches 88.478% top-1 vs 87.472% for Local SGD (+1.006%).
- On CIFAR-100 with ResNet, AdaGK-SGD reaches 60.710% vs 59.240% for Local SGD (+1.470%).
- On ILSVRC2012 with ResNet18, AdaGK-SGD improves Local SGD top-1 from 62.316% to 62.818% (+0.502%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive global knowledge guidance | Local SGD · CIFAR-10, ResNet | TOP-1 accuracy (%): ↑1.006 | Helped |
| Add the MLGK module (ablation) | SlowMo · CIFAR-10, ResNet | TOP-1 accuracy (%): ↑0.275 | Helped |
| Add the MLGK module (ablation) | EASGD · CIFAR-10, ResNet | TOP-1 accuracy (%): ↑0.335 | Helped |
| Add the MLGK module (ablation) | BMUF-Adam · CIFAR-10, ResNet | TOP-1 accuracy (%): ↑0.470 | Helped |
| Add adaptive global knowledge guidance | Local SGD · CIFAR-100, ResNet | TOP-1 accuracy (%): ↑1.470 | Helped |
| Add the MLGK module (ablation) | SlowMo · CIFAR-100, ResNet | TOP-1 accuracy (%): ↑0.195 | Helped |
| Add the MLGK module (ablation) | EASGD · CIFAR-100, ResNet | TOP-1 accuracy (%): ↑0.440 | Helped |
| Add the MLGK module (ablation) | BMUF-Adam · CIFAR-100, ResNet | TOP-1 accuracy (%): ↑0.730 | Helped |
| Add adaptive global knowledge guidance | Local SGD · ILSVRC2012, ResNet18 | TOP-1 accuracy (%): ↑0.502 | Helped |
| Add adaptive global knowledge guidance | Local SGD · CIFAR-100, DenseNet40, batch size 16 | TOP1 test accuracy: ↑0.735 | Helped |
| Add adaptive global knowledge guidance | Local SGD · CIFAR-100, DenseNet40, batch size 32 | TOP1 test accuracy: ↑1.850 | Helped |
| Add adaptive global knowledge guidance | Local SGD · CIFAR-100, DenseNet40, batch size 64 | TOP1 test accuracy: ↑0.608 | Helped |
Surprising findings
- BMUF-Adam, built on Adam, outperforms AdaGK-SGD on single-branch networks, which the authors attribute to Adam being a stronger optimizer than SGD.
Limitations
- Maximum lifetime M is set via a training-loss heuristic rather than the theoretical conditions, which are impractical to compute.
- Experiments are limited to image classification with small ResNet, VGG and DenseNet models and 4 workers.
- Gains are modest (roughly 0.5% to 1.5% top-1) and single-seed-averaged over three runs without reported variance.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ILSVRC2012
Benchmarks: CIFAR-10 CIFAR-100 ILSVRC2012
Metrics: top-1 test accuracy accuracy improvement
Baselines: Local SGD SlowMo EASGD BMUF-Adam
Builds on: Local SGD Elastic Averaging SGD Cooperative SGD SlowMo Time-to-Live
Closest papers by meaning
- STL-SGD: Speeding Up Local SGD with Stagewise Communication Period (AAAI 2021 · similarity 0.89)
- Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity (NeurIPS 2026 · similarity 0.88)
- Distributed Stochastic Optimization via Adaptive SGD (NeurIPS 2018 · similarity 0.88)
- Stability and Generalization of Asynchronous SGD: Sharper Bounds Beyond Lipschitz and Smoothness (NeurIPS 2024 · similarity 0.87)
- Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent (AAAI 2024 · similarity 0.86)
- LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.