Research map / Optimization / Implicit bias and generalization
SGD noise and flat minima
Analyzes how stochastic gradient noise structure and learning rate steer SGD toward flat minima.
112 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Implicit bias and generalization, keyword cluster “sgd · sharpness · noise”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
29 papers in 2023–24 and 15 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 12.3% to 5.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Train faster, generalize better: Stability of stochastic gradient descent (ICML 2016 · 377 citations)
- 2017 · On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · 547 citations)
- 2018 · Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks (ICLR 2018 · 219 citations)
- 2019 · SGD on Neural Networks Learns Functions of Increasing Complexity (NeurIPS 2019 · 96 citations)
- 2020 · Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning (NeurIPS 2020 · 56 citations)
- 2021 · A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima (ICLR 2021 · 38 citations)
- 2022 · Understanding the unstable convergence of gradient descent (ICML 2022 · 8 citations)
- 2023 · Special Properties of Gradient Descent with Large Learning Rates (ICML 2023 · 6 citations)
- 2024 · Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithm (ICML 2024 · 2 citations)
- 2025 · Gaussian Approximation and Concentration of Constant Learning-Rate Stochastic Gradient Descent (NeurIPS 2025 · 1 citation)
- 2026 · Momentum Further Constrains Sharpness at the Edge of Stochastic Stability (ICML 2026 · 0 citations)
Earliest papers here
- Train faster, generalize better: Stability of stochastic gradient descent (ICML 2016 · 377 citations)
- Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks (AAAI 2016 · 156 citations)
- Online ICA: Understanding Global Dynamics of Nonconvex Optimization via Diffusion Processes (NeurIPS 2016)
What has been tried
99 papers here report 718 measured changes (a component added, removed or swapped, compared with a baseline): helped 53%, no clear effect 14%, hurt 24%, mixed 9%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| increase weight decay | Cifar10, Cifar100, ImageNet · convergence | Hurt | Investigating the Role of Weight Decay in Enhancing Nonconvex SGD (CVPR 2025 · 0 citations) | |
| Increase SAM radius from ρ = 0.01 to ρ = 0.1 | Two-layer ReLU network training · Coherence Measure | 121.473 -> 65.656 | Hurt | A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · 0 citations) |
| Increase the free parameter \tilde\kappa | Smallest-eigenvalue subspace · bias decay rate | Helped | Risk Bounds of Accelerated SGD for Overparameterized Linear Regression (ICLR 2024 · 0 citations) | |
| Aggressively decay the learning rate | Walker2d-v4 · rollout reward oscillations | Helped | Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression (ICLR 2024 · 0 citations) | |
| Increase the learning rate from η=0.1 to η=1.2 | Synthetic test data · testing accuracy | 93.75% -> 99.38% | Helped | Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate (ICLR 2024 · 0 citations) |
| Decrease α from 1.9 to 1.75 with Type-I noise | ECG5000, n=10K · pruning ratio | 30.05± 8.12 -> 60.23± 17.85 | Helped | Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD (ICML 2024 · 0 citations) |
| replace the out-of-batch construction with the direct-generalization estimator using K^{-1}A | Huber regression, SGD with batch size n/10 · risk-estimation accuracy | Hurt | Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression (NeurIPS 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 11 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 4 of 11 papers
Baselines
- SGD · 2 of 11 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 44 papers; from OpenAlex and the papers' affiliations).
- University of Chicago · 5 papers
- Massachusetts Institute of Technology · 4 papers
- Inria · 4 papers
- Swiss Federal Institute of Technology Lausanne · 4 papers
- EPFL · 4 papers
- University of California, San Diego · 3 papers
- École Polytechnique · 2 papers
- University of California, Berkeley · 2 papers
- LinkedIn · 2 papers
- Purdue University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Edge-of-stability dynamics (116 papers)
- Stability-based generalization analysis (88 papers)
- Sharpness-aware minimization theory (37 papers)
Elsewhere in the topic
- Shallow ReLU network convergence (132 papers)
- Neural tangent kernel analysis (114 papers)
- Gradient flow and mean-field dynamics (77 papers)
- Low-rank and single-index recovery (70 papers)
- Explicit and implicit regularization interplay (63 papers)
- Implicit regularization of SGD (55 papers)
- Implicit bias in linear networks (53 papers)
- In-context learning as gradient descent (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).