Research map / Learning theory and algorithms / Kernels and regression theory
Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies two-layer mean-field neural networks via a two-timescale limit that reduces training to optimizing an intrinsic kernel, proving convexity, MFLD convergence with discretization error, a Barron-space separation over kernel methods, kernel alignment gains, and a label-noise procedure that reduces degrees of freedom.
Problem
Understanding feature learning in two-layer neural networks in the mean-field regime, specifically whether training learns a data-dependent kernel with global convergence guarantees and sample-complexity advantages over kernel methods.
Approach
Use a two-timescale limit where the second layer is solved optimally, reducing the learning problem to a convex functional over the first-layer distribution; the resulting objective is convex, so mean-field Langevin dynamics converges globally, and label noise adds an implicit degrees-of-freedom regularizer.
Key results
- Two-layer mean-field networks learn a Barron-type (union of RKHS) class with n = O(d log d) samples, while any linear kernel estimator needs n = o(d^m) samples under the stated assumptions.
- At initialization kernel alignment A(mu_0)=O(1/sqrt(d)) and parameter alignment P(mu_0)=O(1/d), whereas for the empirical risk minimizer A and P are Omega(1) for a single-index model.
- The label noise procedure reduces degrees of freedom and improves test loss in the synthetic x1x2 experiment, with linear convergence to the global optimum of the regularized objective.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the first-layer distribution with the mean-field neural network instead of using a fixed kernel | Two-layer mean-field neural network · KL-restricted Barron space | sample complexity: n = O(M); kernel methods require n = Ω(d^m) when M = Ω(d log d) | Helped |
| train the first-layer distribution to adapt the kernel to the target direction | Two-layer mean-field neural network · Single-index model | kernel alignment: O(1/√d) -> Ω(1) | Helped |
| train the first-layer distribution to align parameters with the target direction | Two-layer mean-field neural network · Single-index model | parameter alignment: O(1/d) -> Ω(1) | Helped |
| include intrinsic label noise | Two-layer neural network trained by MFLD · Synthetic regression | degrees of freedom | Hurt |
| add independent label noise during training | Label noise procedure · Synthetic regression | degrees of freedom | Helped |
| add independent label noise during training | Label noise procedure · Synthetic regression | test loss | Helped |
Surprising findings
- Minimizing the limiting functional without regularization tends to increase the degrees of freedom, so intrinsic noise biases the learned kernel toward higher complexity.
Limitations
- LSI constant is exponentially small in 1/lambda, so the convergence rate has exponential computational complexity.
- Two-timescale limit assumes the second layer is solved optimally at each step; discretization analysis is for infinite width with finite-width transfer via a proposition.
- Experiments are limited to a single synthetic task (x1x2) with d=15 and one width.
Experimental setup
Datasets: synthetic data: f(x)=x1x2 in d=15 with Gaussian inputs and bounded noise
Benchmarks: synthetic regression (f(x)=x1x2, d=15)
Metrics: kernel alignment degrees of freedom test loss
Baselines: kernel methods neural tangent kernel
Builds on: mean-field Langevin dynamics Barron space neural tangent kernel two-timescale limit
Closest papers by meaning
- Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks (NeurIPS 2025 · similarity 0.88)
- Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics (ICLR 2025 · similarity 0.88)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.86)
- Asymptotics of feature learning in two-layer networks after one gradient-step (ICML 2024 · similarity 0.86)
- Mean-field Underdamped Langevin Dynamics and its Spacetime Discretization (ICML 2024 · similarity 0.86)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.