Research map / Optimization / Implicit bias and generalization
The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SGD (linear stability, trace of Hessian bounded by 2/eta) instead of GD (only spectral norm bounded) | two-layer ReLU networks and diagonal linear networks · theory and numerical validation | generalization / test error | Helped |
| Increase learning rate for SGD (ablation) | diagonal linear networks · d=1000, n=300, sparse linear target | test error | Helped |
| Increase learning rate for GD (ablation) | diagonal linear networks · d=1000, n=300, sparse linear target | test error | No clear effect |
| Increase learning rate for SGD (ablation) | two-layer ReLU networks · d=100, m=100, n=300 | test performance | Helped |
| Use SGD with fair LR (eta/sqrt(d)) instead of GD with LR eta (ablation) | two-layer ReLU networks · d=100, m=100, n=300 | test performance | Helped |
| Apply gradient clipping during training with large LR | SGD/GD training · numerical experiments | convergence / stability | Helped |
Closest papers by meaning
- Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent (ICML 2020 · similarity 0.94)
- The alignment property of SGD noise and how it helps select flat minima: A stability analysis (NeurIPS 2022 · similarity 0.93)
- A Precise Characterization of SGD Stability Using Loss Surface Geometry (ICLR 2024 · similarity 0.92)
- Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization (NeurIPS 2025 · similarity 0.91)
- On the Unstable Convergence Regime of Gradient Descent (AAAI 2024 · similarity 0.91)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.