Research map / Learning theory and algorithms / Kernels and regression theory
From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops a multi-scale adaptive Bayesian theory of feature learning via a Legendre transform/effective action, unifying kernel rescaling and kernel adaptation theories across mean-field and standard scaling.
Problem
Two theoretical descriptions of feature learning in neural networks (kernel rescaling vs. kernel adaptation) give seemingly contradictory predictions; it is unclear how both can hold and how they relate across scaling regimes.
Approach
Recast the posterior of network outputs as an effective action (Legendre transform of the cumulant-generating function) with a high-dimensional order parameter, evaluated via tree-level (saddle-point) approximation in mean-field scaling and one-loop fluctuation corrections in standard scaling; show that for linear networks the mean output equals that of a rescaled NNGP kernel while covariances and
Key results
- In mean-field scaling, the tree-level adaptive theory accurately predicts train and test outputs of trained linear networks where the NNGP fails.
- For linear networks, the mean predictor equals a rescaled NNGP kernel, but output covariance shows directional adaptation toward teacher directions that rescaling cannot capture.
- For non-linear teacher-student tasks, the adaptive VGA theory predicts the network learns the cubic component with fewer samples, a sample-complexity class change not captured by NNGP or rescaling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-scale adaptive theory instead of the NNGP | Linear networks in mean-field scaling · Ising task | network output prediction accuracy | Helped |
| Add one-loop fluctuation corrections to the tree-level solution (ablation) | Multi-scale adaptive theory · Ising task, standard scaling | test error prediction accuracy | Helped |
| Use the tree-level solution instead of the one-loop solution (ablation) | Multi-scale adaptive theory · Ising task, standard scaling | test error prediction accuracy | Hurt |
| Use adaptive covariance rather than a rescaled NNGP covariance | Linear networks · Teacher-student setting | relative directional feature-learning measure | Helped |
| Use kernel rescaling instead of adaptive covariance | Linear networks · Teacher-student setting | relative directional feature-learning measure | No clear effect |
| Use VGA-based adaptive kernels instead of NNGP or kernel rescaling | Non-linear student network · Non-linear teacher-student setting | learnability of the non-linear component | Helped |
| Use NNGP or kernel rescaling instead of VGA-based adaptive kernels | Non-linear student network · Non-linear teacher-student setting | learnability of the non-linear component | Hurt |
| Use a structural adaptive kernel rather than a rescaled kernel | Non-linear networks · Non-linear teacher-student setting | sample complexity class | Helped |
| Use kernel rescaling instead of structural adaptive kernels | Non-linear networks · Non-linear teacher-student setting | sample complexity class | Hurt |
Surprising findings
- For linear networks, kernel adaptation reduces to an effective isotropic kernel rescaling for the mean output despite qualitatively different kernels.
- Adaptive and rescaling theories disagree already at the level of mean predictions for non-linear networks.
Limitations
- One-loop theory fails to predict empirics in the strongest standard-scaling fluctuation regime, requiring higher-order corrections not implemented.
- Analysis restricted to single hidden-layer networks and linear networks for the main equivalence.
- Input kernel fluctuations are not included in the framework.
Experimental setup
Datasets: Ising task teacher-student synthetic data MNIST
Benchmarks: Ising task teacher-student setting MNIST
Metrics: train and test discrepancies test error directional feature learning measure learnability
Baselines: NNGP kernel rescaling theory tree-level approximation
Builds on: NNGP kernel rescaling theory (Li & Sompolinsky) adaptive kernel theory (Seroussi et al.) Langevin SGD variational Gaussian approximation
Closest papers by meaning
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.90)
- Critical feature learning in deep neural networks (ICML 2024 · similarity 0.90)
- How Feature Learning Can Improve Neural Scaling Laws (ICLR 2025 · similarity 0.90)
- Adaptive kernel predictors from feature-learning infinite limits of neural networks (ICML 2025 · similarity 0.89)
- The Optimization Landscape of SGD Across the Feature Learning Strength (ICLR 2025 · similarity 0.88)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.