Research map / Learning theory and algorithms / Kernels and regression theory

Divergence of Neural Tangent Kernel in Classification Problems

ICLR 2025 · Learning theory and algorithms › Kernels and regression theory · approach: Neural tangent kernel analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that under cross-entropy loss in binary classification, the empirical NTK of wide FCNs and ResNets does not uniformly converge to the NTK over time, because training parameters diverge when the NTK Gram matrix's smallest eigenvalue stays bounded below.

Problem

Whether the empirical neural tangent kernel converges to the NTK during training for classification networks trained with cross-entropy loss, where NTK theory is well established only for MSE regression.

Approach

Proves strict positive definiteness of the NTK for multi-layer FCNs (with biases) and ResNets, then uses a Lyapunov function on output residuals to show parameters diverge under cross-entropy if the empirical NTK's minimum eigenvalue is bounded below; a proof by contradiction then shows the empirical NTK cannot uniformly converge to the NTK over all times.

Key results

• Theorem: for any initialization, there exist x, x' with sup over t of |K_t(x,x') - K_NT(x,x')| >= lambda_0 / (2n^2), where lambda_0 is the minimum eigenvalue of the NTK Gram matrix on training data.
• Synthetic experiment: a width-2000, three-hidden-layer FCN trained 10,000 epochs (lr 0.1) on six circle points with alternating labels shows outputs diverging at all six points.
• MNIST parity experiment: four-layer FCN with width 500, lr 0.5, 100,000 epochs shows empirical NTK values diverging at three selected points.

Measured effects reported in the paper

Surprising findings

• The empirical NTK diverges over training under cross-entropy regardless of network width, contrary to the lazy-regime intuition from regression NTK theory.

Limitations

• The divergence result is qualitative; no divergence rates are provided.
• The argument relies on the assumption that the empirical NTK minimum eigenvalue is bounded below; the paper does not prove this premise holds.
• Experiments are only shown for fully connected networks on small synthetic data and a single MNIST parity task.

Experimental setup

Datasets: MNIST

Benchmarks: MNIST

Metrics: empirical NTK value over training time network output at training points

Builds on: neural tangent kernel kernel logistic regression

Closest papers by meaning

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · similarity 0.90)
• On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.88)
• Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study (NeurIPS 2022 · similarity 0.88)
• Optimized Random Features for the Neural Tangent Kernel (Student Abstract) (AAAI 2025 · similarity 0.88)
• Neural Tangent Kernels Motivate Cross-Covariance Graphs in Neural Networks (ICML 2024 · similarity 0.87)
• Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More (NeurIPS 2025 · similarity 0.86)