Implicit bias and generalization: research map
1,070 papers on Implicit bias and generalization in Optimization, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 3 clusters and 12 approaches. The busiest year so far is 2026.
What it is
Neural networks have far more parameters than data points yet generalise. Part of the reason is that training itself prefers certain solutions (implicit bias). This theory topic studies those preferences: flat minima, the edge of stability, the neural tangent kernel and simplified networks.
Terms you will see on this page:
- Implicit bias
- the kind of solution an algorithm such as SGD tends to find without explicit regularisation.
- Sharpness
- how quickly the loss rises around a solution; flat minima are thought to generalise better.
- Neural tangent kernel (NTK)
- a kernel that describes very wide networks during training.
Start reading with the most cited papers here:
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · 1,345 citations)
- Understanding deep learning requires rethinking generalization (ICLR 2017 · 1,040 citations)
Within Optimization, its share held steady from 27.1% in 2023–24 to 26.7% in 2025–26 (235 → 261 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Implicit bias and generalization in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Edge-of-stability dynamics rising: 11% → 17% of the topic (26 → 45 papers)
- Gradient flow and mean-field dynamics rising: 8% → 10% of the topic (19 → 27 papers)
- Sharpness-aware minimization theory rising: 5% → 7% of the topic (12 → 19 papers)
- SGD noise and flat minima falling: 12% → 6% of the topic (29 → 15 papers)
- Shallow ReLU network convergence falling: 12% → 9% of the topic (28 → 23 papers)
Papers from 2025–2026 that are already cited most:
- Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization (NeurIPS 2025 · 2 citations)
- Convergence Analysis of Federated Learning Methods Using Backward Error Analysis (AAAI 2025 · 2 citations)
- Gaussian Approximation and Concentration of Constant Learning-Rate Stochastic Gradient Descent (NeurIPS 2025 · 1 citations)
Approaches and key papers
Theory of wide and shallow networks · 479 papers
Keywords: relu, width, kernel
Approaches in this cluster:
- Shallow ReLU network convergence (145 papers)
Proves gradient descent convergence and generalization for shallow and deep ReLU or linear networks. - Neural tangent kernel analysis (127 papers)
Uses NTK and linearization theory to analyze training, feature learning, and inductive bias in wide networks. - Gradient flow and mean-field dynamics (83 papers)
Studies training dynamics of wide networks via gradient flow, mean-field limits, and teacher-student models. - Low-rank and single-index recovery (82 papers)
Analyzes gradient descent on matrix sensing, factorization, and single-index models with global guarantees. - In-context learning as gradient descent (42 papers)
Shows transformers implement gradient-descent-like algorithms for in-context linear regression.
Most cited and most cited since 2024:
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · 1,345 citations)
- A Convergence Theory for Deep Learning via Over-Parameterization (ICML 2019 · 688 citations)
- How does PDE order affect the convergence of PINNs? (NeurIPS 2024 · 5 citations)
- Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization (ICLR 2024 · 3 citations)
SGD noise and sharpness · 401 papers
Keywords: sgd, sharpness, noise
Approaches in this cluster:
- SGD noise and flat minima (132 papers)
Analyzes how stochastic gradient noise structure and learning rate steer SGD toward flat minima. - Edge-of-stability dynamics (120 papers)
Characterizes gradient descent at large step sizes where sharpness hovers at the stability threshold. - Stability-based generalization analysis (106 papers)
Bounds generalization of gradient-trained networks through algorithmic stability and Bayesian arguments. - Sharpness-aware minimization theory (43 papers)
Analyzes why and how sharpness-aware minimization works and proposes improved variants.
Most cited and most cited since 2024:
- Understanding deep learning requires rethinking generalization (ICLR 2017 · 1,040 citations)
- Visualizing the Loss Landscape of Neural Nets (NeurIPS 2018 · 606 citations)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · 22 citations)
- Why Warmup the Learning Rate? Underlying Mechanisms and Improvements (NeurIPS 2024 · 15 citations)
Implicit regularization · 190 papers
Keywords: implicit bias, regularization, bias gradient
Approaches in this cluster:
- Explicit and implicit regularization interplay (70 papers)
Studies how weight decay, gradient regularizers, and training dynamics jointly induce implicit regularization. - Implicit regularization of SGD (62 papers)
Analyzes implicit regularization from SGD in least squares, matrix factorization, and diagonal networks. - Implicit bias in linear networks (58 papers)
Characterizes the max-margin-like solutions gradient flow selects in linear and separable classification settings.
Most cited and most cited since 2024:
- Characterizing Implicit Bias in Terms of Optimization Geometry (ICML 2018 · 168 citations)
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (ICML 2018 · 136 citations)
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · 10 citations)
- Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization (NeurIPS 2025 · 2 citations)
Related topics in Optimization
- Distributed and compressed (388)
- Nonconvex and smooth optimization (1,707)
- Optimizers (1,206)
