Research map / Optimization / Implicit bias and generalization
The Break-Even Point on Optimization Trajectories of Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the learning rate from η=0.01 to η=0.1 | SGD training SimpleCNN · CIFAR-10 | max(λ_K^1): 3.30 -> 0.68 | Helped |
| Increase the learning rate from η=0.01 to η=0.1 | SGD training SimpleCNN · CIFAR-10 | max(λ_K*/λ_K^1): 0.24 -> 0.37 | Helped |
| Decrease batch size from S=100 to S=10 | SGD training SimpleCNN · CIFAR-10 | max(Tr(K)): 10.86 -> 5.56 | Helped |
| Increase the learning rate from η=0.01 to η=0.1 | SGD training SimpleCNN · CIFAR-10 | max(λ_H^1): 211.17 -> 26.27 | Helped |
| Use a higher learning rate | SGD training ResNet-32 and SimpleCNN · CIFAR-10 | max(λ_K^1) | Helped |
| Use a smaller batch size | SGD training tested models · Classification tasks studied | max(λ_K^1) | Helped |
| Use a higher learning rate | SGD training tested models · Classification tasks studied | max(λ_K*/λ_K^1) | Helped |
| Use a higher learning rate | SGD training ResNet-32 and SimpleCNN · CIFAR-10 | max(λ_H^1) | Helped |
| Use a higher learning rate | SGD training BERT and DenseNet · MNLI and ImageNet | max(λ_K^1) | Helped |
| Use a higher learning rate with batch normalization | SimpleCNN-BN · CIFAR-10 | conditioning of K | Helped |
| Use low learning rate η=0.001 rather than a high learning rate in the batch-normalized model | SimpleCNN-BN · CIFAR-10 | max(λ_K^1): 12.05 versus 3.30 | Hurt |
| Use low learning rate η=0.001 rather than the comparison setting η=0.01 | SimpleCNN-BN versus SimpleCNN · CIFAR-10 | max(λ_K*/λ_K^1): 0.24 versus 0.343 | Helped |
Closest papers by meaning
- On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length (ICLR 2019 · similarity 0.89)
- A Loss Curvature Perspective on Training Instabilities of Deep Learning Models (ICLR 2022 · similarity 0.88)
- Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate (ICLR 2021 · similarity 0.88)
- Chaotic Dynamics are Intrinsic to Neural Network Training with SGD (NeurIPS 2022 · similarity 0.88)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.88)
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.