Research map / Optimization / Implicit bias and generalization
On the linearity of large non-linear models: when and why the tangent kernel is constant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a linear output layer in a wide network | Wide neural networks · Fixed-radius neighborhood around initialization | Hessian spectral norm: \tilde{O}(1) -> \tilde{O}(1/{\sqrt{m}}) | Helped |
| Add a nonlinear output activation (ablation) | Wide network with linear output layer · Training path as width tends to infinity | Tangent-kernel change: change of the order of \Theta(1) | Hurt |
| Insert a narrow hidden bottleneck (ablation) | Wide network with linear output layer · Training path | Tangent-kernel change | Hurt |
| Increase bottleneck width (ablation) | Wide network with a bottleneck · Synthetic dataset | Tangent-kernel change: \Theta\left(1/\sqrt{m}\right) | Helped |
| Train a nonlinear-output network with gradient descent when |\phi'(z)| is bounded away from zero | Nonlinear-output network · Square-loss optimization | Loss convergence: (1-\eta\mu\rho^2)^t | Helped |
Closest papers by meaning
- Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models (ICLR 2022 · similarity 0.92)
- The asymptotic spectrum of the Hessian of DNN throughout training (ICLR 2020 · similarity 0.91)
- Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture (NeurIPS 2022 · similarity 0.90)
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.89)
- Super Consistency of Neural Network Landscapes and Learning Rate Transfer (NeurIPS 2024 · similarity 0.89)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.