Research map / Optimization / Implicit bias and generalization
On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that under standard (non-mirrored) random initialization, wide-network gradient flow converges uniformly to NTK regression started from a Gaussian process, and that this initialization's bias limits generalization to Omega(n^{-3/(d+3)}) (curse of dimensionality).
Problem
Prior NTK generalization theory assumes mirrored initialization (zero initial output); it is unclear whether standard random initialization changes the generalization of wide networks trained by gradient descent.
Approach
Proves uniform convergence of wide network gradient flow to NTK regression initialized at the Gaussian process limit of the network at initialization, shows the initial output acts as an implicit bias on the regression target, and shows the GP lies in the RKHS interpolation space only for s < 3/(d+1), yielding a lower bound on the generalization rate.
Key results
- Under standard initialization the Gaussian process initial function lies in the RKHS interpolation space [H^NT]^s with probability 1 only for s < 3/(d+1), giving generalization error Omega(n^{-3/(d+3)}) for wide networks trained by GD.
- Estimated smoothness of real goal functions (MNIST 0.40, CIFAR-10 0.09, Fashion-MNIST 0.22) is far above or unrelated to the 3/(d+1) threshold, which the authors argue shows NTK theory cannot explain practice.
- Artificial-data experiments show the empirical learning rate under standard initialization is slower than under mirrored initialization for d=5 and d=10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use standard non-zero initialization instead of mirrored zero-output initialization | Wide neural networks trained in the NTK regime · Generalization | generalization error: n^{-\frac{3}{d+3}} | Hurt |
| Use standard non-zero initialization instead of mirrored zero-output initialization | Neural networks trained with early stopping · Artificial data, d=5 and d=10 | learning rate of generalization error | Hurt |
| Use standard initialization, whose random initial output acts as an implicit bias on the regression function | Kernel gradient flow / wide neural networks · Generalization with smooth regression functions | generalization error: n^{-\frac{3}{d+3}} | Hurt |
| Use standard initialization rather than mirrored initialization | Neural networks in the NTK theory · High-dimensional data | generalization | Hurt |
Surprising findings
- Standard random initialization can make the wide-network generalization rate n^{-3/(d+3)} even for very smooth regression functions, contrary to the minimax-optimal rates under mirrored initialization.
- The reported real-data smoothness values (e.g., CIFAR-10 0.09) are small, suggesting real data is not as smooth as assumed, which conflicts with the authors' own claim that real smoothness far exceeds the threshold.
Limitations
- Lower bound is proved for spherical data with uniform measure and a modified network without the first-layer bias.
- Experiments are small-scale with artificial data and smoothness estimates from one-hidden-layer NTK; the paper does not train large networks.
Experimental setup
Datasets: MNIST CIFAR-10 Fashion-MNIST artificial regression data
Benchmarks: MNIST CIFAR-10 Fashion-MNIST
Metrics: L2 generalization error estimated smoothness of goal function log-error vs log n slope (learning rate)
Baselines: mirrored initialization
Builds on: Neural Tangent Kernel kernel gradient flow mirrored initialization analysis (Hu et al.; Suzuki et al.)
Closest papers by meaning
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · similarity 0.93)
- Gradient Descent in Neural Networks as Sequential Learning in Reproducing Kernel Banach Space (ICML 2023 · similarity 0.91)
- Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization (ICML 2022 · similarity 0.90)
- What can linearized neural networks actually say about generalization? (NeurIPS 2021 · similarity 0.90)
- Optimized Random Features for the Neural Tangent Kernel (Student Abstract) (AAAI 2025 · similarity 0.90)
- Divergence of Neural Tangent Kernel in Classification Problems (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.