Research map / Optimization / Implicit bias and generalization

Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Gradient flow and mean-field dynamics · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves that regularized Newton training of overparameterized single-hidden-layer networks converges in probability to a deterministic limit (via a Newton neural tangent kernel) with explicit rates, and that the limit converges linearly to zero loss without spectral bias.

Problem

Convergence of second-order (regularized Newton) training of neural networks in the overparameterized (infinite-width) limit, where the implicit Newton update involves a potentially indefinite Hessian and a linear system whose dimension grows with width.

Approach

Uses a block-diagonal-plus-low-rank structure and the push-through (Woodbury) identity to write the Newton update explicitly, defining a Newton neural tangent kernel (NNTK) that converges to a deterministic limit; with regularizer scaled as gamma/N^{2beta-1}, the NNTK eigenvalues are lower-bounded, removing spectral bias.

Key results

• Under the stated assumptions, the finite-width trajectory converges in probability to the infinite-width limit at rate O(N^{-(beta-1/2)} + N^{-(1-beta)} + log(2M)/N^{1/2}) (up to delta factors).
• In the infinite-width limit, the residual satisfies ||f_k - y||_2 <= (1 - alpha*lambda_min(NNTK))^k ||y||_2, with rate uniform across frequencies for small gamma.
• With gamma=0 and alpha=1, the limit NNTK equals identity and the unregularized Newton method reaches the target in one step.

Surprising findings

• The unregularized Newton method converges to the target in a single step in the infinite-width limit.

Limitations

• Analysis restricted to single-hidden-layer networks with bounded activations; deep and unbounded activations left for future work.
• Does not show quadratic convergence; only linear convergence in the limit.
• Analysis covers the exact regularized Newton method, not quasi-Newton, K-FAC, Shampoo or stochastic/mini-batch variants.

Experimental setup

Datasets: synthetic 1D regression targets (e.g., 2x+0.4sin(5πx), 4x+sin(2πx)+0.8(...))

Benchmarks: synthetic 1D regression with high-frequency components

Metrics: training loss NNTK eigenvalues distance between finite-width and limit NNTK

Baselines: gradient descent Adam Gauss-Newton method

Builds on: neural tangent kernel Gram-Gauss-Newton trust-region methods lazy training

Closest papers by meaning

• Fast Convergence of Natural Gradient Descent for Over-Parameterized Neural Networks (NeurIPS 2019 · similarity 0.91)
• Geometric Convergence of Gauss–Newton for Neural Networks: Riemannian Geometry and Adaptive Damping (ICML 2026 · similarity 0.90)
• An Improved Analysis of Training Over-parameterized Deep Neural Networks (NeurIPS 2019 · similarity 0.90)
• Fast Convergence of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks (ICLR 2026 · similarity 0.90)
• Rethinking Gauss-Newton for learning over-parameterized models (NeurIPS 2023 · similarity 0.90)
• Quantitative convergence of trained neural networks to Gaussian processes (NeurIPS 2025 · similarity 0.88)