Research map / Learning theory and algorithms / Kernels and regression theory
Neural Networks as Kernel Learners: The Silent Alignment Effect
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train from small initialization on whitened data so the kernel aligns before its scale grows | Homogeneous neural networks · Whitened data | agreement between final network and final-NTK regression | Helped |
| Use the final NTK instead of the initial NTK | Small-init ReLU MLP · Whitened MNIST | agreement with neural network predictions | Helped |
| Use the final NTK instead of the initial NTK | ReLU network · Whitened CIFAR-10 | agreement with neural network predictions | Helped |
| Whiten the input data (ablation) | Small-init ReLU network · CIFAR-10 | agreement with final-NTK predictor | Helped |
| Use more anisotropic, less-whitened input data (ablation) | Small-init ReLU network · CIFAR-10 | agreement with final-NTK predictor | Hurt |
| Replace whitened data with unwhitened data (ablation) | Neural networks · Unwhitened data | silent alignment | Hurt |
| Increase network depth (ablation) | Deep linear network · Whitened synthetic data | final kernel-target alignment | Helped |
| Use small initialization | Homogeneous neural networks · Whitened data | separation of kernel alignment from loss decrease | Helped |
| Use the final NTK rather than the initial NTK | Wide-ResNet · Whitened CIFAR-10 | agreement with neural network predictions | Helped |
| Increase feature-learning strength A (ablation) | Linear kernel for transfer learning · Transfer tasks correlated with the learned direction | generalization error | Helped |
Closest papers by meaning
- On the Target-kernel Alignment: a Unified Analysis with Kernel Complexity (NeurIPS 2024 · similarity 0.88)
- From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning (ICML 2025 · similarity 0.88)
- How Graph Neural Networks Learn: Lessons from Training Dynamics (ICML 2024 · similarity 0.88)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.88)
- Generalization in Kernel Regression Under Realistic Assumptions (ICML 2024 · similarity 0.87)
- Neural Tangent Kernels Motivate Cross-Covariance Graphs in Neural Networks (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.