Research map / Learning theory and algorithms / Kernels and regression theory
Quadratic models for understanding catapult dynamics of neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Neural Quadratic Models (second-order Taylor expansions of shallow ReLU networks) exhibit catapult dynamics at large learning rates and that their generalization parallels wide networks' in the catapult regime.
Problem
Linear (NTK) approximations of wide neural networks cannot capture the catapult phase and the large-learning-rate generalization benefits of finite-width networks; the question is which models and conditions produce catapult dynamics.
Approach
Use a second-order Taylor expansion (Neural Quadratic Model) of a shallow ReLU network as an analytically tractable surrogate; prove that its gradient descent dynamics have sub-critical, catapult, and divergent regimes, and show empirically that its test loss tracks that of the original networks while linearized models diverge.
Key results
- Theory: for a single training example, NQM gradient descent with learning rate in [2/λ0, 4/λ0] has a catapult where loss grows to order m(ηλ0−2)^2/log m before decreasing to O(1), while rates above 4/λ0 diverge.
- Empirically, NQMs and wide networks have nearly identical test loss at sub-critical learning rates, and both show improved test loss at super-critical learning rates where the linearized model diverges.
- Training with large learning rates decreases the top eigenvalues of the tangent kernel, as the catapult occurs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with a super-critical learning rate instead of a sub-critical learning rate | Neural Quadratic Model (NQM) · Single training example | training loss: increases to the order of Ω(m(ηλ₀−2)²/log m) then decreases to O(1) | Mixed |
| Train with a super-critical learning rate instead of a sub-critical learning rate | Neural Quadratic Model (NQM) · Multiple unidimensional training examples | projected training loss: increases to the order of Ω(m(ηλᵢ(0)−2)²/log m) then decreases to O(1) | Mixed |
| Train with super-critical rather than sub-critical learning rates | Neural Quadratic Model (NQM) and corresponding neural network · CIFAR-2, MNIST, SVHN, FSDD, and AG NEWS | best test loss | Helped |
| Train with super-critical learning rates | Linearized neural network · CIFAR-2, MNIST, SVHN, FSDD, and AG NEWS | training loss | Hurt |
| Use the quadratic model rather than the linearized model | Neural network approximation · CIFAR-2, MNIST, SVHN, FSDD, and AG NEWS | best test loss | Helped |
| Vary learning rate within the sub-critical regime | NQM, corresponding neural network, and linear model · CIFAR-2, MNIST, SVHN, FSDD, and AG NEWS | test loss | No clear effect |
| Train with a super-critical learning rate, causing the catapult phase | Neural Quadratic Model (NQM) · Single training example | tangent-kernel top eigenvalue | Helped |
Surprising findings
- The transition from non-linear to linear dynamics is discontinuous in width: the catapult domain does not shrink to a point as width grows, for both networks and NQMs.
Limitations
- Theoretical catapult analysis is for shallow two-layer ReLU NQMs, a single training example, or unidimensional inputs.
- Generalization claims are empirical and rely on comparing best test loss over learning rates.
Experimental setup
Datasets: CIFAR-2 MNIST SVHN Free Spoken Digit Dataset AG NEWS
Benchmarks: CIFAR-2 MNIST SVHN Free Spoken Digit Dataset AG NEWS
Metrics: best test loss best test accuracy training loss largest tangent kernel eigenvalue
Baselines: linearized neural network (f_lin) wide neural networks (f)
Builds on: Neural Quadratic Models neural tangent kernel catapult phase analysis (Lewkowycz et al. 2020)
Closest papers by meaning
- Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws (NeurIPS 2025 · similarity 0.86)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.85)
- Quantitative convergence of trained neural networks to Gaussian processes (NeurIPS 2025 · similarity 0.84)
- Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning (ICML 2024 · similarity 0.84)
- Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks (NeurIPS 2025 · similarity 0.84)
- Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.