Research map / Learning theory and algorithms / Kernels and regression theory
Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives high-probability lower and upper bounds on the smallest NTK eigenvalue for randomly initialized ReLU networks on arbitrary sphere data that is delta-separated, using a hemisphere transform and spherical harmonics, with no distributional assumptions and constant input dimension allowed.
Problem
Existing lower bounds on the smallest NTK eigenvalue need distributional assumptions on the data (Lipschitz concentration) and only hold w.h.p. when input dimension grows with sample size (d0 ~ log n or sqrt n). The paper targets bounds for arbitrary data on a sphere with fixed input dimension.
Approach
Measure collinearity via delta-separation (distance between points and their reflections), exchange Gaussian weights for uniform-sphere expectations using scale invariance/homogeneity, interpret the infinite-width NTK as a hemisphere transform, and lower bound its norm via Bessel's inequality with spherical harmonics and the addition formula, with a matching upper bound from closest-pair gradient
Key results
- Shallow ReLU networks: if d1 = Omega~(||X||^2 d0^3 delta^-2) then w.h.p. lambda_min(K) = Omega~(d0^-3 delta^2) and lambda_min(K) = O(delta') where delta' is the minimum pairwise distance.
- Deep pyramidal ReLU networks with d1 = Omega~(n d0^3 delta^-4) give lambda_min(K) = Omega~(d0^-3 delta^4) and upper bound O(L) w.h.p.
- For uniform data on the sphere with d0 d1 = Omega~(n^{1+4/(d0-1)}), lambda_min(K) = Omega~(n^{-4/(d0-1)}) and O~(n^{-2/(d0-1)}), tight up to logs when d0 = Omega(log n).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use δ-separation instead of distributional assumptions on the data | Shallow ReLU NTK analysis · Arbitrary δ-separated spherical data | smallest NTK eigenvalue lower bound: \lambda_{\min}(\mK) \gtrsim \left(1 + \frac{d\log(1/\delta)}{\log(d)}\right)^{-3} \delta^2 | Helped |
| Allow the input dimension to remain constant as sample size grows | Proposed shallow and deep NTK bounds · δ-separated spherical data | smallest NTK eigenvalue lower bound | Helped |
| Specialize the shallow bound to iid uniform sphere data | Shallow ReLU NTK · Uniform sphere data | smallest NTK eigenvalue lower bound: \tilde{\Omega}(n^{-4/(d_0 - 1)}) | Helped |
Limitations
- Bounds only hold for the ReLU activation (homogeneity needed).
- Upper bound for deep networks does not capture data separation.
- Requires data normalized to lie on the unit sphere.
Experimental setup
Metrics: smallest eigenvalue of NTK (theoretical bounds)
Baselines: Bombari et al. 2023 shallow NTK bounds Nguyen 2021 deep NTK bounds Nguyen & Mondelli 2020 Montanari & Zhong 2022
Builds on: neural tangent kernel Du et al. NTK convergence analysis Bombari et al. NTK eigenvalue bounds Nguyen et al. deep NTK conditioning Funk-Hecke formula
Closest papers by meaning
- Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability (NeurIPS 2025 · similarity 0.85)
- Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More (NeurIPS 2025 · similarity 0.84)
- Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel (ICML 2025 · similarity 0.84)
- Neural Tangent Kernels Motivate Cross-Covariance Graphs in Neural Networks (ICML 2024 · similarity 0.84)
- Optimized Random Features for the Neural Tangent Kernel (Student Abstract) (AAAI 2025 · similarity 0.83)
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.