Research map / Optimization / Implicit bias and generalization
A generalized neural tangent kernel for surrogate gradient learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes the neural tangent kernel to surrogate gradient learning (SG-NTK), shows the direct NTK for sign activations diverges on the diagonal, and gives an SG-NTK for sign networks that predicts SGL in finite-width experiments.
Problem
Surrogate gradient learning for networks with non-differentiable (step-like) activations such as sign lacks theoretical foundation, and the NTK framework does not directly apply because the activation derivative is ill-defined.
Approach
Define a generalized NTK using quasi-Jacobian matrices in which the true activation derivative is replaced by a surrogate derivative, prove its convergence at initialization and during training in the infinite-width limit, and derive a finite analytic SG-NTK for the sign activation via erf approximations.
Key results
- Empirical NTKs of erf_m networks converge to the analytic NTK as width grows, and the analytic NTK diverges on the diagonal as m grows (singular kernel).
- Empirical SG-NTKs converge to the analytic SG-NTK at initialization and after training, and the analytic SG-NTK converges to a finite limit as m goes to infinity.
- Mean of 500 sign-activation networks trained with SGL at width 500 is close to the SG-NTK GP mean, with most networks inside the confidence band; agreement already holds at width 100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the SG-NTK instead of the classical NTK for surrogate gradient learning | Sign-activation networks trained with SGL · Unit-sphere regression | agreement with kernel prediction | Helped |
| Increase hidden-layer width | Sign-activation networks trained with SGL · Unit-sphere regression | agreement with SG-NTK prediction | Helped |
| Approximate sign activation with increasingly steep scaled error functions | Classical NTK · Unit-sphere regression | kernel diagonal finiteness | Hurt |
| Use the SG-NTK with sign activation instead of the classical NTK | Sign-activation networks trained with SGL · Unit-sphere regression | kernel limit finiteness | Helped |
| Compare empirical kernels at increasing hidden-layer widths with their analytic limits | Networks with erf_m activation · Unit-sphere regression | empirical-to-analytic NTK agreement | Helped |
| Increase activation steepness m | Networks trained with gradient descent · Unit-sphere regression | empirical-to-analytic NTK convergence | Hurt |
| Compare empirical SG-NTKs at increasing hidden-layer widths with their analytic limits | Networks trained with SGL and erf_m activation · Unit-sphere regression | empirical-to-analytic SG-NTK agreement | Helped |
Surprising findings
- The direct NTK for the sign activation is singular: its diagonal diverges, so kernel regression interpolates training points and outputs zero off-training-set.
- The SG-NTK for sign activation is finite despite the sign derivative being a Dirac delta, and it predicts SGL dynamics even though the convergence during training is only proven for differentiable activations.
Limitations
- Results apply only to sufficiently wide networks with random initialization under the NTK framework.
- Convergence of the SG-NTK during training is proven only for activations with well-defined derivatives; the sign case is supported only by simulations.
- Experiments are limited to a small 2D regression problem.
Experimental setup
Datasets: synthetic 2D regression on unit circle, 15 training points, target f(x,y)=4xy^2-0.8x^3+1.2y^2-0.8x^2y
Benchmarks: synthetic regression on unit circle
Metrics: mean squared error between empirical and analytic NTK/SG-NTK agreement of SGL network distribution with SG-NTK GP mean an
Baselines: direct NTK with erf approximations kernel regression with Sigma_sign
Builds on: neural tangent kernel Gaussian process limit of wide networks (Neal / Lee et al.) Jacot et al. NTK convergence theorem Yang tensor programs
Closest papers by meaning
- Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks (ICML 2023 · similarity 0.89)
- Gradient Descent in Neural Networks as Sequential Learning in Reproducing Kernel Banach Space (ICML 2023 · similarity 0.89)
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.87)
- Optimized Random Features for the Neural Tangent Kernel (Student Abstract) (AAAI 2025 · similarity 0.87)
- The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training (AAAI 2025 · similarity 0.86)
- Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.