Research map / Learning theory and algorithms / Kernels and regression theory
Adaptive kernel predictors from feature-learning infinite limits of neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives two adaptive, data-dependent kernel predictors (aNBK from a feature-learning Bayesian posterior, aNTK from DMFT fixed points of gradient flow with weight decay) for infinitely wide MLPs and CNNs in the rich mu-P regime, with numerical solvers and comparisons to lazy NNGP/NTK kernels.
Problem
Infinitely wide neural networks in the lazy regime are kernel machines with fixed kernels, which do not capture feature learning; it is unclear whether rich-regime infinite-width networks admit kernel predictors and what their kernels are.
Approach
Two infinite-width limits of rich-regime networks (noisy gradient flow posterior sampling, and gradient flow with weight decay via DMFT fixed points) yield representer-theorem kernel predictors whose kernels adapt deterministically to the training data; the Bayesian one is a min-max saddle point problem, and deep linear cases reduce to scalar overlaps.
Key results
- On a two-layer MLP trained on two CIFAR10 classes, the adaptive kernels (aNTK, aNBK) achieve lower test loss than the lazy NNGPK predictor for larger sample sizes P, with a gap for feature learning strength gamma_0=0.3.
- For deep linear networks with whitened data, the last-layer overlap scales as c_L ~ 1 + L gamma_0^2 for small L gamma_0^2 and as c_L ~ L gamma_0^2 / ln(L gamma_0^2) for large depth, matching simulations.
- Finite-width (N=1028 to 5000) trained networks are closely matched by the theoretical adaptive kernel predictors, including non-Gaussian pre-activation densities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive feature-learning kernels (aNBK/aNTK) instead of lazy NNGP/NTK predictors | Two-layer MLP on CIFAR10 airplane vs automobile · CIFAR10 airplane vs automobile | test loss | Helped |
| Use adaptive feature-learning kernels instead of lazy predictors | Two-layer MLP on CIFAR10 airplane vs automobile · CIFAR10 airplane vs automobile, small sample size | test loss | Mixed |
| Use a two-layer CNN instead of a two-layer MLP | DMFT kernel predictors at the same feature-learning strength · CIFAR10 animate/inanimate data, large sample size | test loss | Helped |
| Increase feature-learning strength γ₀ | Bayesian deep MLP adaptive kernels · CIFAR10 | kernel alignment with label covariance | Helped |
| Add weight decay to gradient-flow training | Infinite-width feature-learning networks · Rich-regime convergence | agreement with aNTK predictor | Helped |
| Remove weight decay (set λ = 0) (ablation) | Infinite-width feature-learning networks · Rich-regime convergence | agreement with aNTK predictor | Hurt |
Surprising findings
- With lambda=0 (no weight decay), the network predictor at convergence does not match the aNTK kernel predictor, since initial-condition terms persist late in training.
- In the small-P regime, lazy NNGP predictors can outperform feature-learning predictors because feature learning overfits the test points.
Limitations
- Solving the min-max saddle point problem for deep nonlinear networks is computationally hard and needs importance sampling and warm starts.
- Experiments are limited to small datasets (P up to about 1000) and shallow or moderate-depth networks, with finite-width agreement shown at N=1028 to 5000.
- The predictors are derived in an infinite-width mu-P limit and for lambda>0 weight decay; the lambda=0 case does not give a kernel predictor.
Experimental setup
Datasets: CIFAR10 MNIST
Benchmarks: CIFAR10 MNIST
Metrics: test loss kernel alignment pre-activation density
Baselines: NNGPK lazy predictor NNGP posterior mean NTK lazy predictor trained finite-width network
Builds on: NNGP neural tangent kernel maximal update parameterization DMFT for gradient flow training
Closest papers by meaning
- Critical feature learning in deep neural networks (ICML 2024 · similarity 0.89)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.89)
- Deep Kernel Posterior Learning under Infinite Variance Prior Weights (ICLR 2025 · similarity 0.88)
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.87)
- How Feature Learning Can Improve Neural Scaling Laws (ICLR 2025 · similarity 0.87)
- An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.