Research map / Optimization / Implicit bias and generalization
Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the exact discrete-time SGD covariance instead of the continuous-time approximation | SGD with white noise · 1D quadratic loss | variance prediction agreement | Helped |
| Use the exact discrete-time SGD covariance instead of the continuous-time approximation | SGD with white noise · 2D quadratic loss | parameter fluctuation prediction agreement | Helped |
| Use the exact discrete-time SGD covariance instead of the continuous-time approximation | SGD with minibatch noise · 1D linear regression | variance prediction agreement | Helped |
| Use the exact discrete-time SGD covariance instead of the continuous-time approximation | SGD with minibatch noise · 2D linear regression | parameter fluctuation prediction agreement | Helped |
| Approximate minibatch noise covariance as proportional to the Hessian | Discrete-time SGD theory · Minibatch-noise experiments | variance prediction agreement | Hurt |
| Use the discrete-time momentum theory instead of the continuous-time approximation | SGD with momentum and white noise · 1D quadratic loss with λk = 2.75 | variance prediction agreement | Helped |
| Use the discrete-time momentum theory instead of the continuous-time approximation | SGD with momentum and minibatch noise · Linear regression with λk = 2.75 | variance prediction agreement | Helped |
| Use the discrete-time escaping-efficiency prediction instead of the continuous-time prediction | SGD · Sharp-minimum escape experiment | escaping efficiency prediction agreement | Helped |
| Use the discrete-time Kramers escape-rate approximation instead of the continuous-time prediction | SGD · Escape experiment with rescaled loss | escape rate trend | Helped |
| Use a discrete-time SGD theory that permits finite-covariance non-Gaussian noise | SGD · 1D quadratic loss with Student's t and centered χ² noise | variance prediction agreement | Helped |
Closest papers by meaning
- Asymptotic theory of SGD with a general learning-rate (NeurIPS 2025 · similarity 0.93)
- Strength of Minibatch Noise in SGD (ICLR 2022 · similarity 0.92)
- Global Convergence and Stability of Stochastic Gradient Descent (NeurIPS 2022 · similarity 0.91)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.90)
- Statistical Guarantees for High-Dimensional Stochastic Gradient Descent (NeurIPS 2025 · similarity 0.89)
- Nonasymptotic Analysis of Stochastic Gradient Descent with the Richardson–Romberg Extrapolation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.