Research map / Optimization / Implicit bias and generalization
On the Inductive Bias of Neural Tangent Kernels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train all layers instead of only the last layer (ablation) | Two-layer ReLU network · Even Lipschitz functions on the sphere | approximation rate: ([?]delta/eta)^{-1/(p/2-1)} log(delta/eta) vs. (delta/eta)^{-1/(p/2)} | Helped |
| Train all layers instead of only the last layer (ablation) | Two-layer ReLU network · ReLU NTK RKHS on the sphere | smoothness of kernel mapping: 1-Lipschitz vs. not Lipschitz | Hurt |
| Use the full ReLU NTK rather than the last-layer-only κ₁ kernel (ablation) | Two-layer ReLU network · Unit-sphere inputs | kernel-mapping smoothness: sqrt(||x - y||) vs. ||x - y|| | Hurt |
| Use ReLU rather than a twice-differentiable activation with bounded derivative moments | Two-layer network NTK · Unit-sphere inputs | kernel-mapping smoothness: sqrt(||x - y||) vs. ||x - y|| | Hurt |
| Use exponential activation σ(u) = e^{u-2} | Two-layer network NTK · Unit-sphere inputs | kernel-mapping Lipschitz constant: sqrt(3) | Helped |
| Use soft-plus activation σ(u) = log(1 + e^u) | Two-layer network NTK · Unit-sphere inputs | kernel-mapping Lipschitz constant: ≈ 1.75 | Helped |
| Use the full convolutional ReLU NTK rather than the last-layer-only CKN kernel (ablation) | CNN with two hidden convolutional layers · Infinite MNIST digits and deformations | growth of relative feature distance under small deformations: square-root growth rate rather than a linear one | Hurt |
| Use small patches | Convolutional ReLU NTK · Deformation stability | stability bound | Helped |
| Add an initial anti-aliasing pooling layer | Convolutional ReLU NTK · Deformation stability | deformation stability | Helped |
Closest papers by meaning
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · similarity 0.90)
- The asymptotic spectrum of the Hessian of DNN throughout training (ICLR 2020 · similarity 0.89)
- Gradient Descent in Neural Networks as Sequential Learning in Reproducing Kernel Banach Space (ICML 2023 · similarity 0.89)
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.88)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.87)
- Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.