Research map / Optimization / Nonconvex and smooth optimization
AdaGrad Stepsizes: Sharp Convergence Over Nonconvex Landscapes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdaGrad-Norm (single adaptive stepsize) instead of fixed-stepsize SGD | SGD on smooth nonconvex / linear regression · Synthetic Gaussian linear regression | gradient norm convergence across b0 | Helped |
| Use AdaGrad-Norm instead of SGD with fixed/decaying stepsize | deep networks · MNIST, CIFAR10, ImageNet | test accuracy sensitivity to b0 | Helped |
| Use AdaGrad-Norm with very small initial b0 instead of SGD | deep networks · MNIST, CIFAR10 | test accuracy | No clear effect |
| Large initial b0 (small effective stepsize) for AdaGrad-Norm | AdaGrad-Norm on linear regression · Synthetic Gaussian linear regression | gradient norm convergence speed | No clear effect |
| Per-neuron / per-channel stepsize (instead of strict global norm) in neural network AdaGrad-Norm implementation | AdaGrad-Norm on ResNet and MLPs · image classification | generalization | No clear effect |
Closest papers by meaning
- High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize (ICLR 2022 · similarity 0.92)
- Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction (NeurIPS 2024 · similarity 0.91)
- Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness (ICLR 2025 · similarity 0.90)
- AdaGrad under Anisotropic Smoothness (ICLR 2025 · similarity 0.89)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.88)
- Directional Smoothness and Gradient Methods: Convergence and Adaptivity (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.