Research map / Optimization / Implicit bias and generalization
Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically shows that oscillating SGD with large learning rates learns weak features under a feature-noise model with strong and weak features, improving generalization over small-learning-rate SGD, which fails on weak-only test data.
Problem
Why does training with large learning rates (SGD oscillation) improve neural network generalization compared with small learning rate SGD, which converges smoothly?
Approach
Oscillation of the network output around the label in large-learning-rate SGD accumulates linearly rather than cancelling, which drives learning of weak (low-norm, always-present) features that small-learning-rate SGD fails to learn because it fits the strong features first.
Key results
- On synthetic data, large learning rate (eta=1.2) SGD reaches 99.38% test accuracy versus 93.75% for small learning rate (eta=0.1), averaged over 5 seeds.
- Theory shows large learning rate oscillating SGD learns the weak signal to a constant scale delta, while small learning rate SGD keeps weak signal at initialization scale (~sigma_0 ||v||).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the learning rate from η=0.1 to η=1.2 (ablation) | Two-layer CNN trained with SGD · Synthetic test data | testing accuracy: 93.75% -> 99.38% | Helped |
| Use the large learning rate η=1.2 instead of the small learning rate η=0.1 (ablation) | Two-layer CNN trained with SGD · Synthetic weak-signal data | weak signal learning | Helped |
Surprising findings
- The sum of fluctuations of the network output around the label accumulates linearly rather than cancelling, which is the mechanism for weak feature learning.
Limitations
- Analysis restricted to a two-layer CNN with ReLU^2 activation, a simplified feature-noise data model, and a specific oscillation pattern (Assumption on oscillation magnitude).
- Experiments are small-scale synthetic settings plus one ResNet-18 CIFAR-10 illustration; no large-scale validation.
- Requires conditions on the learning rate and initialization scale; the oscillation assumption is only sufficient, not necessary.
Experimental setup
Datasets: CIFAR-10 synthetic feature-noise data
Benchmarks: synthetic feature-noise test data CIFAR-10
Metrics: test accuracy training loss signal learning strength
Baselines: SGD with small learning rate
Builds on: edge of stability feature learning theory of deep learning (Allen-Zhu and Li) harmful overfitting under low SNR
Closest papers by meaning
- Special Properties of Gradient Descent with Large Learning Rates (ICML 2023 · similarity 0.89)
- Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks (NeurIPS 2023 · similarity 0.89)
- On the Generalization Benefit of Noise in Stochastic Gradient Descent (ICML 2020 · similarity 0.88)
- On the generalization of learning algorithms that do not converge (NeurIPS 2022 · similarity 0.88)
- Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup (NeurIPS 2019 · similarity 0.88)
- Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.