Research map / Learning theory and algorithms / Kernels and regression theory
What can linearized neural networks actually say about generalization?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the network, allowing its empirical NTK to rotate | MLP on CIFAR2 · CIFAR2 | energy concentration on top K=50 eigenvectors: 26.0% -> 63.6% | Helped |
| Train the network, allowing its empirical NTK to rotate | LeNet on CIFAR2 · CIFAR2 | energy concentration on top K=50 eigenvectors: 25.8% -> 83.0% | Helped |
| Train the network, allowing its empirical NTK to rotate | ResNet18 on CIFAR2 · CIFAR2 | energy concentration on top K=50 eigenvectors: 22.7% -> 96.7% | Helped |
| Use kernel alignment at initialization to order tasks by difficulty | Neural networks and their linearized models · Binarized NTK eigenfunctions on CIFAR10 | relative learning complexity | Helped |
| Use a linearized model instead of the nonlinear network | Models trained on binarized NTK eigenfunctions · Binarized NTK eigenfunctions on CIFAR10 | validation accuracy | Helped |
| Use neural-network-derived NADs to choose linear predictor directions | CNNs · Linearly separable predictor tasks | classification performance | Helped |
| Increase the training set size | Neural networks compared with their linear approximations · CIFAR2 | test accuracy relative to the linear approximation | Helped |
| Linearize at the trained network's kernel instead of at initialization | ResNet18 · Binarized 400th NTK eigenfunction | convergence speed | Helped |
| Linearize at the trained network's kernel instead of at initialization | ResNet18 · Binarized 400th NTK eigenfunction | test accuracy | Hurt |
| Pretrain the nonlinear network before extracting the linearized model | Linearized networks · Binarized 400th NTK eigenfunction | test accuracy | Hurt |
| Replace SGD with Adam | Neural networks and linearized models · Binarized NTK eigenfunctions on CIFAR10 | validation accuracy trend across eigenfunction index | No clear effect |
| Replace CIFAR10 support with MNIST | Neural networks and linearized models · Binarized NTK eigenfunctions on MNIST | relative advantage across tasks | Mixed |
Closest papers by meaning
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.90)
- Disentangling Trainability and Generalization in Deep Neural Networks (ICML 2020 · similarity 0.90)
- Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks (ICLR 2020 · similarity 0.90)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks (NeurIPS 2018 · similarity 0.89)
- Optimization and Adaptive Generalization of Three layer Neural Networks (ICLR 2022 · similarity 0.89)
- On the linearity of large non-linear models: when and why the tangent kernel is constant (NeurIPS 2020 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.