Research map / Optimization / Implicit bias and generalization
Neural tangent kernel analysis
Uses NTK and linearization theory to analyze training, feature learning, and inductive bias in wide networks.
114 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Implicit bias and generalization, keyword cluster “relu · width · kernel”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
29 papers in 2023–24 and 29 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 12.3% to 11.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · 1,345 citations)
- 2019 · Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path? (ICML 2019 · 88 citations)
- 2020 · Neural Tangents: Fast and Easy Infinite Neural Networks in Python (ICLR 2020 · 92 citations)
- 2021 · Deep Frequency Principle Towards Understanding Why Deeper Learning Is Faster (AAAI 2021 · 44 citations)
- 2022 · Projective Manifold Gradient Layer for Deep Rotation Regression (CVPR 2022 · 18 citations)
- 2023 · Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network (ICML 2023 · 7 citations)
- 2024 · How does PDE order affect the convergence of PINNs? (NeurIPS 2024 · 5 citations)
- 2025 · Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · 2 citations)
- 2026 · A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights (ICML 2026 · 0 citations)
Earliest papers here
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · 1,345 citations)
- On the Information Bottleneck Theory of Deep Learning (ICLR 2018 · 515 citations)
- The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network (NeurIPS 2018 · 41 citations)
What has been tried
84 papers here report 574 measured changes (a component added, removed or swapped, compared with a baseline): helped 58%, no clear effect 11%, hurt 22%, mixed 8%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| use finest variable splitting (|ξ| = 1, p = 2) instead of coarser splitting (|ξ| = 2, p = 3) | Fourth-order elastic beam equation · training loss reduction | Helped | How does PDE order affect the convergence of PINNs? (NeurIPS 2024 · 5 citations) | |
| Increase ReLU network depth | Synthetic, MNIST, FashionMNIST, SVHN, Librispeech · NTK condition number | Helped | Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · 2 citations) | |
| Give covariance and cumulant spikes a finite overlap u·v = ρ, while keeping latent variables uncorrelated | MCM · speed of learning the non-Gaussian part | No clear effect | Sliding Down the Stairs: How Correlated Latent Variables Accelerate Learning with Neural Networks (ICML 2024 · 2 citations) | |
| Shifted-ReLU experiments: Setting 2 (H1 + H2/sqrt2 target) vs Setting 1 (linear only) across log(eta)/log(n) | synthetic, n=1000, N=500, d=300 · train/test error vs log(eta)/log(n) | Mixed | A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks (ICML 2024 · 2 citations) | |
| Set the last-layer inference step-size exponent to 0 instead of -1 | FashionMNIST · inference stability across widths | Hurt | Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation (ICLR 2025 · 1 citation) | |
| Replace the isotropic prior with a source condition, 1<b<a+1 | Power-law spectrum, source condition · population risk above irreducible risk | M^{1-b} + (N_eff γ)^{(1-b)/a} + min{M,(N_eff γ)^{1/a}}/N_eff | Mixed | Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · 1 citation) |
| Use no regularization instead of weight decay or AGOP regularization | Modular addition, p = 61 · test accuracy | does not go to 100% | Hurt | Emergence in non-neural models: grokking modular arithmetic via average gradient outer product (ICML 2025 · 0 citations) |
| add IDIZC to the identity-only initialization | CIFAR-10 · accuracy | 87.01 -> 92.9 | Helped | IDInit: A Universal and Stable Initialization Method for Neural Network Training (ICLR 2025 · 0 citations) |
| Add a learning-rate warm-up phase | CINIC-10, 50 epochs · validation accuracy | Helped | On the Weight Dynamics of Deep Normalized Networks (ICML 2024 · 0 citations) | |
| use Tanh activation | CIFAR-10 · feature diversity as width increases | Hurt | Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025) | |
| use Softplus instead of ReLU | Neural ODE training · convergence speed | Helped | Global Convergence in Neural ODEs: Impact of Activation Functions (ICLR 2025) |
What reviewers will expect
Datasets and baselines named in the 20 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MNIST · 6 of 20 papers
- CIFAR-10 · 6 of 20 papers
- CIFAR-100 · 2 of 20 papers
- FashionMNIST · 2 of 20 papers
- SVHN · 2 of 20 papers
- SST2 · 2 of 20 papers
Baselines
- gradient descent · 2 of 20 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 58 papers; from OpenAlex and the papers' affiliations).
- Stanford University · 3 papers
- Swiss Federal Institute of Technology Lausanne · 3 papers
- Microsoft · 2 papers
- Tsinghua University · 2 papers
- SISSA · 2 papers
- University of California, Los Angeles · 2 papers
- EPFL · 2 papers
- Harvard University · 2 papers
- University of Pennsylvania · 2 papers
- Seoul National University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Shallow ReLU network convergence (132 papers)
- Gradient flow and mean-field dynamics (77 papers)
- Low-rank and single-index recovery (70 papers)
- In-context learning as gradient descent (41 papers)
Elsewhere in the topic
- Edge-of-stability dynamics (116 papers)
- SGD noise and flat minima (112 papers)
- Stability-based generalization analysis (88 papers)
- Explicit and implicit regularization interplay (63 papers)
- Implicit regularization of SGD (55 papers)
- Implicit bias in linear networks (53 papers)
- Sharpness-aware minimization theory (37 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).