Research map / Optimization / Implicit bias and generalization
The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use anisotropic noise aligned with the Hessian instead of isotropic noise | Gradient descent with noise · 2-D toy example | success rate of escaping to the flat minimum | Helped |
| Use GLD 1st eigvec(H) noise instead of isotropic noise | Gradient descent with noise · 2-D toy example | success rate of escaping to the flat minimum | Helped |
| Use GLD diagonal noise instead of Hessian-aligned anisotropic noise | Gradient descent with noise · 2-D toy example | success rate of escaping to the flat minimum | Hurt |
| Use SGD noise instead of full-batch gradient descent | LeNet-like network · FashionMNIST with corrupted labels | test accuracy | Helped |
| Use Hessian-aligned anisotropic noise instead of isotropic noise | Gradient descent with noise · FashionMNIST with corrupted labels | test accuracy | Helped |
| Use GLD dynamic isotropic noise instead of curvature-aligned anisotropic noise | Gradient descent with noise · FashionMNIST with corrupted labels | escape from the sharp minimum | Hurt |
| Use GLD diagonal noise instead of curvature-aligned anisotropic noise | Gradient descent with noise · FashionMNIST with corrupted labels | escape from the sharp minimum | Hurt |
| Use GLD constant isotropic noise instead of curvature-aligned anisotropic noise | Gradient descent with noise · FashionMNIST with corrupted labels | escape from the sharp minimum | Hurt |
| Use GLD Hessian noise | Gradient descent with noise · FashionMNIST with corrupted labels | escape from the sharp minimum | Helped |
| Use GLD leading covariance noise | Gradient descent with noise · FashionMNIST with corrupted labels | escape from the sharp minimum | Helped |
| Use GLD 1st eigvec(H) noise | Gradient descent with noise · FashionMNIST with corrupted labels | escape from the sharp minimum | Helped |
| Use SGD instead of isotropic-noise dynamics | VGG-11 · SVHN | test accuracy | Helped |
Closest papers by meaning
- Strength of Minibatch Noise in SGD (ICLR 2022 · similarity 0.91)
- A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima (ICLR 2021 · similarity 0.90)
- Generalization of noisy SGD in unbounded non-convex settings (ICML 2025 · similarity 0.89)
- Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.89)
- Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise (NeurIPS 2025 · similarity 0.89)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.