Research map / Learning theory and algorithms / Kernels and regression theory
Neural tangent kernel analysis
Study wide networks via the neural tangent kernel and feature-learning limits.
109 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Kernels and regression theory, keyword cluster “kernels · hilbert · reproducing”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
26 papers in 2023–24 and 30 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 10.5% to 12.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Exponential expressivity in deep neural networks through transient chaos (NeurIPS 2016 · 290 citations)
- 2017 · Deep Information Propagation (ICLR 2017 · 24 citations)
- 2018 · Demystifying MMD GANs (ICLR 2018 · 171 citations)
- 2019 · Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality (ICLR 2019 · 105 citations)
- 2020 · Deep Learning for Handling Kernel/model Uncertainty in Image Deconvolution (CVPR 2020 · 64 citations)
- 2021 · A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels (ICLR 2021 · 20 citations)
- 2022 · Rethinking Influence Functions of Neural Networks in the Over-Parameterized Regime (AAAI 2022 · 10 citations)
- 2023 · Symmetric Pruning in Quantum Neural Networks (ICLR 2023 · 8 citations)
- 2024 · PINNACLE: PINN Adaptive ColLocation and Experimental points selection (ICLR 2024 · 6 citations)
- 2025 · How Feature Learning Can Improve Neural Scaling Laws (ICLR 2025 · 2 citations)
- 2026 · Pruning at Initialisation through the lens of Graphon Limit: Convergence, Expressivity, and Generalisation (ICML 2026 · 0 citations)
Earliest papers here
- Exponential expressivity in deep neural networks through transient chaos (NeurIPS 2016 · 290 citations)
- Deep Information Propagation (ICLR 2017 · 24 citations)
- Learning Infinite Layer Networks Without the Kernel Trick (ICML 2017 · 3 citations)
What has been tried
86 papers here report 647 measured changes (a component added, removed or swapped, compared with a baseline): helped 64%, no clear effect 11%, hurt 17%, mixed 8%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Remove weight decay (set λ = 0) | Rich-regime convergence · agreement with aNTK predictor | Hurt | Adaptive kernel predictors from feature-learning infinite limits of neural networks (ICML 2025 · 1 citation) | |
| Use A1 random features without Gibbs sampling | CT · MSE | 199.15 -> 183.63 | Helped | Optimized Random Features for the Neural Tangent Kernel (Student Abstract) (AAAI 2025 · 0 citations) |
| Use least-squares coefficient calculation instead of the inner-product method | Transfer benchmarks · performance | Helped | Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces (ICML 2025 · 0 citations) | |
| Add one-loop fluctuation corrections to the tree-level solution | Ising task, standard scaling · test error prediction accuracy | Helped | From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · 0 citations) | |
| add batch normalization layers | PDE benchmarks · test error | No clear effect | A Bregman Proximal Viewpoint on Neural Operators (ICML 2025 · 0 citations) | |
| Change LayerNorm placement among every layer, first layer only, and second layer only | UCI Protein OOD · R² | 0.50 ± 0.02; 0.50 ± 0.02; 0.50 ± 0.01 | No clear effect | Just One Layer Norm Guarantees Stable Extrapolation (NeurIPS 2025 · 0 citations) |
| Use stronger shortcut correlation | Dataset with varied shortcut-label correlation · availability | Helped | Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More (NeurIPS 2025 · 0 citations) | |
| use cosine-normalized trace NTK instead of unnormalized trNTK⁰ | CIFAR10 poisoned-data experiment, clean test · Kendall-τ | 0.344 -> 0.643 | Helped | Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models (ICLR 2024 · 0 citations) |
| Set damping coefficient γ=√L instead of γ=1 | Continuous-time convergence · energy-gap decay rate | O(\mathscr{C}_{\textsf{LSI}}/\mathscr{L}) -> O(\mathscr{C}_{\textsf{LSI}}/\sqrt{\mathscr{L}}) | Helped | Mean-field Underdamped Langevin Dynamics and its Spacetime Discretization (ICML 2024 · 0 citations) |
| replace Euclidean similarity with Manhattan similarity | capacity, memorization criterion; MNIST, CIFAR10, and Tiny ImageNet · fraction correctly retrieved | 0.624 ± 0.22 -> 0.793 ± 0.14; 0.223 ± 0.19 -> 0.502 ± 0.24; 0.223 ± 0.21 -> 0.669 ± 0.24 | Helped | Relating Hopfield Networks to Episodic Control (NeurIPS 2024 · 0 citations) |
| vary polynomial order Nγ | Jetplane image reconstruction · PSNR (dB) | Mixed | Learning Input Encodings for Kernel-Optimal Implicit Neural Representations (ICML 2025) | |
| Remove the learnable grid contribution from MPE | Image regression on ImageNet synonym sets · NTK eigenvalue spectrum | Hurt | How Learnable Grids Recover Fine Detail in Low Dimensions: A Neural Tangent Kernel Analysis of Multigrid Parametric Encodings (ICLR 2025) |
What reviewers will expect
Datasets and baselines named in the 22 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MNIST · 5 of 22 papers
- CIFAR10 · 2 of 22 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 56 papers; from OpenAlex and the papers' affiliations).
- Harvard University · 3 papers
- The University of Tokyo · 3 papers
- University of California, San Diego · 3 papers
- University of California, Los Angeles · 2 papers
- National Institute of Informatics · 2 papers
- The Graduate University for Advanced Studies · 2 papers
- New York University · 2 papers
- NTT · 2 papers
- RIKEN · 2 papers
- University of Helsinki · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- RKHS kernel methods (127 papers)
- Kernel regression under shift (89 papers)
- Kernel two-sample testing (30 papers)
- Random feature approximation (36 papers)
Elsewhere in the topic
- Statistical perspectives on generalization (99 papers)
- Information-theoretic generalization bounds (94 papers)
- High-dimensional asymptotic generalization (82 papers)
- Generalization theory of deep networks (67 papers)
- Empirical generalization measures (61 papers)
- High-dimensional linear regression theory (61 papers)
- Kernel ridge regression analysis (57 papers)
- Double descent and overfitting (53 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).