Research map / Architectures and efficiency / RNNs and normalization
The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
Wei Hu, Lechao Xiao, Ben Adlam, Jeffrey Pennington
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Approximate early training of the two-layer network with gradient descent on its corresponding linear model | Two-layer fully connected network · Synthetic Gaussian data | prediction discrepancy | No clear effect |
| Continue training the network after the linear model reaches its optimum | Two-layer fully connected network · Synthetic Gaussian data | training progress: 1,000 steps | Mixed |
| Increase the input dimension | Two-layer network and corresponding linear model · Synthetic Gaussian data | prediction-discrepancy MSE | Improved |
| Include the norm-dependent feature in the corresponding linear model (ablation) | Two-layer ReLU network · Synthetic norm-dependent target | approximation to network predictions | Improved |
| Use a naive linear model with the norm-feature coefficients set to zero (ablation) | Corresponding linear model for a two-layer ReLU network · Synthetic norm-dependent target | approximation to network predictions | Hurt |
| Approximate early training of the network with its corresponding linear model | Four-hidden-layer fully connected network and CNN · Downsampled CIFAR-10 cats vs. horses | test loss: 1,000 steps | No clear effect |
| Continue training beyond the early linear-learning phase | Four-hidden-layer fully connected network and CNN · Downsampled CIFAR-10 cats vs. horses | test loss in the complement of the input span: 1,000 steps | Improved |
| Approximate early training with the corresponding linear model | Four-hidden-layer CNN · Full-size CIFAR-10 binary classification | early-time agreement | Mixed |
| Bound the initialization-kernel discrepancy in spectral norm rather than relying on Frobenius norm | Two-layer network NTK analysis · Synthetic Gaussian data | kernel-discrepancy scaling with dimension: d^{-1.263} vs. d^{-0.718} | Improved |
Closest papers by meaning
- Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel (NeurIPS 2020 · similarity 0.90)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.89)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.88)
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.88)
- Understanding and Leveraging the Learning Phases of Neural Networks (AAAI 2024 · similarity 0.87)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
