Research map / Optimization / Nonconvex and smooth optimization
Escaping Saddles with Stochastic Gradients
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Require stochastic gradients to have nonzero second moment along the minimum-curvature direction (CNC), rather than isotropic noise | CNC-PGD and CNC-SGD · Smooth non-convex objectives | dimension dependence of convergence: free | Helped |
| Use vanilla SGD without explicitly added perturbation, with periodic step-size increases | CNC-SGD · Smooth non-convex objectives under CNC | second-order stationarity: ${\cal O}(\epsilon^{-10})$ iterations in the contribution statement | Helped |
| Use stochastic gradients rather than isotropic noise to perturb GD | CNC-PGD · Smooth non-convex objectives under CNC | second-order stationarity: ${\cal O}(\epsilon^{-2})$ iterations in the contribution statement | Helped |
| Use stochastic gradients on learning half-spaces with the stated loss and unit-sphere support conditions | Stochastic gradients · Learning half-spaces | second moment projected onto a negative-curvature eigenvector: $(\lambda/c)^2$ | Helped |
| Increase isotropic-noise perturbation across dimensions | Isotropic noise · Neural networks with increasing width or depth | variance along the most-negative-curvature eigenvector: ${\cal O}(1/d)$ | Hurt |
| Use stochastic gradients as network width and depth increase | Stochastic gradients · MNIST neural networks | variance component along the most-negative-curvature direction | No clear effect |
| Consider eigenvectors associated with larger-magnitude eigenvalues | Stochastic gradients · MNIST neural networks | projected gradient variance | Helped |
Closest papers by meaning
- First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time (NeurIPS 2018 · similarity 0.91)
- Taming Stochastic Gradient Descent: Almost Sure Convergence and Saddle-Point Avoidance under $(L_{0},L_{1})$-Smoothness (ICML 2026 · similarity 0.91)
- The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects (ICML 2019 · similarity 0.90)
- How to Escape Saddle Points Efficiently (ICML 2017 · similarity 0.90)
- Dealing With Unbounded Gradients in Stochastic Saddle-point Optimization (ICML 2024 · similarity 0.88)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.