Research map / Optimization / Nonconvex and smooth optimization
SGDR: Stochastic Gradient Descent with Warm Restarts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the default schedule with cosine warm restarts | WRN-28-10 · CIFAR-10 | test error (%): 4.13 -> 4.03 | Helped |
| Replace the default schedule with cosine warm restarts | WRN-28-10 · CIFAR-100 | test error (%): 20.21 -> 19.58 | Helped |
| Use a fixed 200-epoch period instead of doubling periods (ablation) | SGDR on WRN-28-10 · CIFAR-10 | test error (%): 4.03 -> 3.86 | Helped |
| Use a fixed 200-epoch period instead of doubling periods (ablation) | SGDR on WRN-28-10 · CIFAR-100 | test error (%): 19.58 -> 19.98 | Hurt |
| Start with a 1-epoch period and double periods (ablation) | SGDR on WRN-28-10 · CIFAR-10 and CIFAR-100 | test error (%): 4.13 -> 4.09; 20.21 -> 19.74 | Mixed |
| Start with a 10-epoch period and double periods (ablation) | SGDR on WRN-28-10 · CIFAR-10 and CIFAR-100 | test error (%): 4.13 -> 4.03; 20.21 -> 19.58 | Helped |
| Widen the network from WRN-28-10 to WRN-28-20 with SGDR | SGDR · CIFAR-10 and CIFAR-100 | test error (%): 4.03 -> 3.74; 19.58 -> 18.70 | Helped |
| Ensemble three SGDR snapshots from one run instead of using one snapshot (ablation) | SGDR on WRN-28-10 · CIFAR-10 and CIFAR-100 | test error (%): 4.03 -> 3.51; 19.57 -> 17.75 | Helped |
| Use three independent runs with three snapshots each (ablation) | SGDR ensemble on WRN-28-10 · CIFAR-10 and CIFAR-100 | test error (%): 3.51 -> 3.25; 17.75 -> 16.64 | Helped |
| Increase the ensemble from three to 16 runs, retaining three snapshots per run (ablation) | SGDR ensemble on WRN-28-10 · CIFAR-10 and CIFAR-100 | test error (%): 3.25 -> 3.14; 16.64 -> 16.21 | Helped |
| Take snapshots near the final epochs rather than before restarts (ablation) | SGDR ensemble on WRN-28-10 · CIFAR-10 and CIFAR-100 | test error | No clear effect |
| Use SGDR instead of the default schedule | WRN-28-10 · Downsampled ImageNet | anytime performance | Helped |
Closest papers by meaning
- Using Statistics to Automate Stochastic Optimization (NeurIPS 2019 · similarity 0.87)
- Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (NeurIPS 2019 · similarity 0.86)
- Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks (ICLR 2018 · similarity 0.86)
- Incremental Gradient Descent with Small Epoch Counts is Surprisingly Slow on Ill-Conditioned Problems (ICML 2025 · similarity 0.86)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.85)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.