Research map / Optimization / Optimizers
Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use stagewise SGD instead of SGD with iterative theoretical step decay | SGD on ResNet-20 · CIFAR-10 and CIFAR-100, with and without regularization | testing error (%): 16.25 -> 8.34; 19.18 -> 9.01; 43.51 -> 32.25; 45.78 -> 34.95 | Helped |
| Use stagewise SGD instead of SGD with fixed-frequency heuristic decay | SGD on ResNet-20 · CIFAR-10 and CIFAR-100, with and without regularization | testing error (%): 8.34 -> 8.34; 10.81 -> 9.01; 33.67 -> 32.25; 37.19 -> 34.95 | Mixed |
| Use stagewise SHB instead of SHB with iterative theoretical step decay | SHB on ResNet-20 · CIFAR-10 and CIFAR-100, with and without regularization | testing error (%): 15.67 -> 8.30; 16.55 -> 8.61; 39.15 -> 32.85; 46.23 -> 34.49 | Helped |
| Use stagewise SHB instead of SHB with fixed-frequency heuristic decay | SHB on ResNet-20 · CIFAR-10 and CIFAR-100, with and without regularization | testing error (%): 8.58 -> 8.30; 10.28 -> 8.61; 33.30 -> 32.85; 37.56 -> 34.49 | Helped |
| Use stagewise SNAG instead of SNAG with iterative theoretical step decay | SNAG on ResNet-20 · CIFAR-10 and CIFAR-100, with and without regularization | testing error (%): 17.64 -> 8.00; 16.76 -> 8.93; 39.34 -> 31.42; 44.21 -> 33.29 | Helped |
| Use stagewise SNAG instead of SNAG with fixed-frequency heuristic decay | SNAG on ResNet-20 · CIFAR-10 and CIFAR-100, with and without regularization | testing error (%): 8.85 -> 8.00; 10.34 -> 8.93; 33.89 -> 31.42; 36.84 -> 33.29 | Helped |
| Use stagewise AdaGrad instead of AdaGrad with iterative theoretical step decay | AdaGrad on ResNet-20 · CIFAR-10 and CIFAR-100, with and without regularization | testing error (%): 12.11 -> 9.09; 13.96 -> 9.51; 39.09 -> 33.95; 44.49 -> 34.62 | Helped |
| Use stagewise AdaGrad instead of AdaGrad with fixed-frequency heuristic decay | AdaGrad on ResNet-20 · CIFAR-10 and CIFAR-100, with and without regularization | testing error (%): 10.71 -> 9.09; 13.80 -> 9.51; 37.04 -> 33.95; 41.06 -> 34.62 | Helped |
| Use stagewise AdaGrad instead of AMSGrad | Adaptive optimization on ResNet-20 · CIFAR-10 and CIFAR-100, with and without regularization | testing error (%): 10.76 -> 9.09; 11.13 -> 9.51; 38.62 -> 33.95; 39.96 -> 34.62 | Helped |
| Use stagewise SGD rather than fixed-frequency heuristic decay | SGD on CIFAR-10 and CIFAR-100 · Training error | training error | Mixed |
| Use stagewise step-size schedules rather than iteratively decreasing step sizes | SGD, SHB, and SNAG · CIFAR-10 and CIFAR-100 | testing error | Helped |
| Use stagewise algorithms rather than heuristic fixed-frequency decay | SGD, SHB, SNAG, and AdaGrad · CIFAR-10 and CIFAR-100 | testing error | Mixed |
Closest papers by meaning
- SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variance (ICML 2023 · similarity 0.92)
- An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Bias (NeurIPS 2021 · similarity 0.91)
- Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction (NeurIPS 2024 · similarity 0.90)
- Random Scaling and Momentum for Non-smooth Non-convex Optimization (ICML 2024 · similarity 0.89)
- Universal Gradient Methods for Stochastic Convex Optimization (ICML 2024 · similarity 0.89)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.