Research map / Optimization / Implicit bias and generalization
An Alternative View: When Does SGD Escape Local Minima?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Analyze SGD as optimizing the loss convolved with its gradient-noise distribution | SGD on smooth non-convex objectives satisfying convolved one-point strong convexity · Theoretical convergence | squared distance to target: with probability at least 1/2, ||y_t-x^*||_2^2 <= O(log(T_2) eta r^2/c) | Helped |
| Use a large step size before shrinking it to a smaller step size | SGD training of smaller ResNet and DenseNet on CIFAR-10 and CIFAR-100 · CIFAR-10, CIFAR-100 | validation loss | Helped |
| Use a large step size before shrinking it to a smaller step size | SGD training of smaller ResNet and DenseNet on CIFAR-10 and CIFAR-100 · CIFAR-10, CIFAR-100 | distance of local minimum from initialization | Mixed |
| Use a noise level that is neither too small nor too large | SGD on the spiky motivating function · Spiky function | convergence to a local region near the target | Helped |
| Increase noise from a level that is too small | SGD on the spiky motivating function · Spiky function | convergence to a local region near the target | Helped |
| Increase noise beyond an appropriate level | SGD on the spiky motivating function · Spiky function | convergence to a local region near the target | Hurt |
| Shrink the learning rate after reaching a local region | SGD on the spiky motivating function · Spiky function | distance to target | Helped |
| Use a learning rate that is too large | Gradient descent on a one-point-convex function · Theoretical one-step result | distance to target: eta > 2c'||x_t-x^*||_2^2 / ||nabla f(x_t)||_2^2 | Hurt |
| Use VGG instead of ResNet or DenseNet | Neural-network loss-surface neighborhood analysis · CIFAR-10, CIFAR-100 | one-point convexity in trajectory neighborhoods | Hurt |
Closest papers by meaning
- SGD Can Converge to Local Maxima (ICLR 2022 · similarity 0.91)
- SGD Converges to Global Minimum in Deep Learning via Star-convex Path (ICLR 2019 · similarity 0.90)
- Benign Underfitting of Stochastic Gradient Descent (NeurIPS 2022 · similarity 0.90)
- SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs (NeurIPS 2021 · similarity 0.89)
- SGD and Hogwild! Convergence Without the Bounded Gradients Assumption (ICML 2018 · similarity 0.89)
- A Precise Characterization of SGD Stability Using Loss Surface Geometry (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.