Research map / Optimization / Implicit bias and generalization
Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that loss spikes in SGD are catapults occurring in the top eigenspace of the NTK, and that smaller batch sizes induce more catapults, which improves AGOP alignment and test performance through feature learning.
Problem
Why do spikes in the training loss occur when training neural networks with SGD at large learning rates, and how do these spikes relate to generalization (why small-batch SGD generalizes better)?
Approach
Connects SGD loss spikes to catapult dynamics from GD (occurring in the top NTK eigenspace with decreasing NTK spectral norm), and argues catapults improve generalization by increasing alignment between the trained model's AGOP and the true AGOP; batch-dependent critical learning rates make smaller batches induce more catapults.
Key results
- Match rate between loss increases in SGD and the condition eta > critical learning rate of the batch is 97.32% (shallow net), 96.17% (5-layer FCN), and 94.67% (5-layer CNN).
- On a synthetic dataset, decreasing SGD batch size from 2000 (GD) to 5 raises AGOP alignment from 0.81 to 0.95 and lowers test loss from 0.74 to 0.42.
- Loss spikes in SGD occur in the top-1 eigenspace of the NTK, and each spike coincides with a significant drop in the NTK spectral norm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decrease SGD batch size from 2000 (GD) to 5 | SGD on the synthetic dataset · Synthetic dataset | AGOP alignment: 0.81 -> 0.95 | Helped |
| decrease SGD batch size from 2000 (GD) to 5 | SGD on the synthetic dataset · Synthetic dataset | Test loss: 0.74 -> 0.42 | Helped |
| decrease batch size | SGD with a large learning rate · Rank-2 regression, Rank-3 regression, SVHN, and CelebA | Test loss/error | Helped |
| decrease batch size | SGD with a sufficiently small learning rate · Rank-2 regression, Rank-3 regression, SVHN, and CelebA | Generalization performance | No clear effect |
| increase the number of catapults by repeatedly increasing the learning rate during training | GD · Rank-2 regression, Rank-3 regression, SVHN, and CelebA | Test loss/error | Helped |
Surprising findings
- With small learning rates (critical LR/40), batch size has little effect on test performance, indicating the generalization gain depends on catapults.
- Multiple catapults in GD let training converge at a learning rate that would diverge if used from the start, implying a smaller final NTK norm.
Limitations
- Relies on experiments rather than theory for the SGD catapult and generalization claims.
- AGOP alignment uses a state-of-the-art model as a substitute for the true model on real datasets.
- Small training sets (128 to 2000 points) and wide-network/NTK-regime settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-2 SVHN CelebA Fashion-MNIST USPS synthetic polynomial regression (rank-2, rank-3, rank-4)
Benchmarks: CIFAR-10 SVHN CelebA Fashion-MNIST USPS synthetic rank-2/3/4 regression
Metrics: training loss decomposition spectral norm of NTK number of catapults AGOP alignment (cosine similarity) test loss test error
Baselines: Gradient descent with small learning rate Adagrad Adadelta Adam RMSprop SGD with momentum
Builds on: Catapult dynamics (Lewkowycz et al. 2020) Neural Tangent Kernel Average Gradient Outer Product (AGOP) feature learning
Closest papers by meaning
- Does SGD really happen in tiny subspaces? (ICLR 2025 · similarity 0.88)
- Learning Curves for SGD on Structured Features (ICLR 2022 · similarity 0.88)
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.88)
- Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation (NeurIPS 2026 · similarity 0.87)
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · similarity 0.87)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.