Research map / Optimization / Implicit bias and generalization

Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks

NeurIPS 2025 · Optimization › Implicit bias and generalization · approach: Neural tangent kernel analysis · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows analytically and empirically that ReLU nonlinearity in wide networks increases pairwise model-gradient angles and reduces NTK condition number, with more gain at greater depth, implying faster gradient-descent convergence.

Problem

How nonlinear activations affect the geometry of model gradient features and the conditioning of the neural tangent kernel in wide networks, relative to linear networks.

Approach

Compares ReLU networks against their activation-free linear counterparts at infinite width, showing ReLU increases the model-gradient angle between similar inputs (better separation), which yields a smaller NTK condition number that decreases with depth; in the infinite-width-then-depth limit the NTK condition number converges to (n+4)/3.

Key results

• ReLU networks of every depth L=1..10 show larger minimum pairwise model-gradient angle and smaller NTK condition number than matched linear networks, with condition number decreasing monotonically in depth.
• In the infinite-width-then-depth limit, the model-gradient angle converges to arccos(1/4) (about 75.5 degrees) for all non-parallel inputs, and the NTK condition number converges to (n+4)/3.
• Deeper ReLU networks converge faster in training loss on MNIST, FashionMNIST and Librispeech, but MNIST test accuracy drops at depth 8 and beyond (97.57% at L=6 to 97.19% at L=12).

Measured effects reported in the paper

Surprising findings

• Test accuracy on MNIST decreases at large depth despite faster optimization, suggesting an optimization-generalization trade-off.
• In the infinite-depth limit, all non-parallel inputs become equally separated (about 75.5 degrees) regardless of their original similarity, which may yield trivial generalization.

Limitations

• Theoretical analysis covers only ReLU; other activations are supported only numerically.
• Analysis is in the infinite-width NTK regime with first layer trainable and other layers fixed in the main theorem.
• Convergence benefit is shown empirically and via worst-case bounds, not a tight measured rate.

Experimental setup

Datasets: synthetic Gaussian MNIST FashionMNIST SVHN Librispeech

Benchmarks: MNIST FashionMNIST SVHN Librispeech

Metrics: minimum pairwise model gradient angle NTK condition number training loss test accuracy

Baselines: linear neural network

Builds on: neural tangent kernel linear neural network implicit preconditioning

Closest papers by meaning

• The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training (AAAI 2025 · similarity 0.89)
• A generalized neural tangent kernel for surrogate gradient learning (NeurIPS 2024 · similarity 0.87)
• Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More (NeurIPS 2025 · similarity 0.87)
• ReLUs Are Sufficient for Learning Implicit Neural Representations (ICML 2024 · similarity 0.86)
• Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.86)
• DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.86)