Research map / Optimization / Implicit bias and generalization
Stability-based generalization analysis
Bounds generalization of gradient-trained networks through algorithmic stability and Bayesian arguments.
88 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Implicit bias and generalization, keyword cluster “sgd · sharpness · noise”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
14 papers in 2023–24 and 15 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 6.0% to 5.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Matrix Completion has No Spurious Local Minimum (NeurIPS 2016 · 179 citations)
- 2017 · Understanding deep learning requires rethinking generalization (ICLR 2017 · 1,040 citations)
- 2018 · Understanding Batch Normalization (NeurIPS 2018 · 208 citations)
- 2019 · Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks (NeurIPS 2019 · 121 citations)
- 2020 · Deep Double Descent: Where Bigger Models and More Data Hurt (ICLR 2020 · 503 citations)
- 2021 · Multiple Descent: Design Your Own Generalization Curve (NeurIPS 2021 · 26 citations)
- 2022 · Assessing Generalization of SGD via Disagreement (ICLR 2022 · 19 citations)
- 2023 · Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization (ICLR 2023 · 6 citations)
- 2024 · Neglected Hessian component explains mysteries in sharpness regularization (NeurIPS 2024 · 2 citations)
- 2025 · A Method for Enhancing Generalization of Adam by Multiple Integrations (AAAI 2025 · 1 citation)
- 2026 · Revisiting the Volume Hypothesis (ICML 2026 · 0 citations)
Earliest papers here
- Matrix Completion has No Spurious Local Minimum (NeurIPS 2016 · 179 citations)
- Deep Learning without Poor Local Minima (NeurIPS 2016 · 22 citations)
- Understanding deep learning requires rethinking generalization (ICLR 2017 · 1,040 citations)
What has been tried
69 papers here report 581 measured changes (a component added, removed or swapped, compared with a baseline): helped 60%, no clear effect 10%, hurt 23%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add second-derivative information to ReLU (augmented ReLU) | ImageNet · test accuracy | Helped | Neglected Hessian component explains mysteries in sharpness regularization (NeurIPS 2024 · 2 citations) | |
| Use second-order instead of first-order multiple integration (MIAdam2) | CIFAR-10 and CIFAR-100 classification · top-1 test accuracy | ResNet50 CIFAR-10: 94.17 (± 0.10) -> 94.28 (± 0.15); DenseNet121 CIFAR-100: 77.02 (±0.10) -> 76.21 (±0.33) | Mixed | A Method for Enhancing Generalization of Adam by Multiple Integrations (AAAI 2025 · 1 citation) |
| Increase gating depth from D=2 to D=4 | VGG-16 (CIFAR-10) · theoretical speed-up within 5% / 10% of max. test accuracy | 16.13 / 52.70 -> 18.88 / 48.97 | Mixed | Differentiable Sparsity via $D$-Gating: Simple and Versatile Structured Penalization (NeurIPS 2025 · 1 citation) |
| Replace Softmax with StableMax in the attention mechanism | WikiText-103 · Top-5 accuracy | 60.48% ± 0.04 -> 58.52% ± 0.04 | Hurt | Grokking at the Edge of Numerical Stability (ICLR 2025 · 1 citation) |
| Upsample selected examples with USEFUL instead of random examples | CIFAR10 and CIFAR100 · test error | Helped | Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · 1 citation) | |
| add acceleration term a_m to m | CIFAR10, batch 16384 · top-1 accuracy (%) | 87.06 -> 88.83 | Helped | Exploring Landscapes for Better Minima along Valleys (NeurIPS 2025 · 0 citations) |
| Standard initialization scale with sigmoid output vs large initialization | MNIST generalization speed · steps to generalization | 10^2 instead of 10^5 steps | Helped | Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond (NeurIPS 2024 · 0 citations) |
| Reduce the Gauss-Newton learning rate | CIFAR-10, mini-batch · training and test loss | No clear effect | Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 10 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 6 of 10 papers
- CIFAR-100 · 4 of 10 papers
- MNIST · 3 of 10 papers
- ImageNet-1k · 2 of 10 papers
Baselines
- Adam · 2 of 10 papers
- AdamW · 2 of 10 papers
- SGD · 2 of 10 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 29 papers; from OpenAlex and the papers' affiliations).
- University of California, Los Angeles · 3 papers
- Engineering Research Center of Intelligent Technology for Agriculture · 2 papers
- Key Laboratory of Smart Farming for Agricultural Animals · 2 papers
- Xi’an Jiaotong University · 2 papers
- Mohamed bin Zayed University of Artificial Intelligence · 2 papers
- Huazhong Agricultural University · 2 papers
- Microsoft Research Asia · 2 papers
- The University of Hong Kong · 2 papers
- Massachusetts Institute of Technology · 2 papers
- Google · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- SGD noise and flat minima (112 papers)
- Edge-of-stability dynamics (116 papers)
- Sharpness-aware minimization theory (37 papers)
Elsewhere in the topic
- Shallow ReLU network convergence (132 papers)
- Neural tangent kernel analysis (114 papers)
- Gradient flow and mean-field dynamics (77 papers)
- Low-rank and single-index recovery (70 papers)
- Explicit and implicit regularization interplay (63 papers)
- Implicit regularization of SGD (55 papers)
- Implicit bias in linear networks (53 papers)
- In-context learning as gradient descent (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).