Research map / Optimization / Implicit bias and generalization
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a reduced learning rate along the top K Hessian directions (NSGD, γ=0.01) rather than SGD with η=0.01 | Resnet-32 · CIFAR-10 | test accuracy: 86.4% -> 87.0% | Helped |
| Reduce the learning rate along the top K Hessian directions | Resnet-32 · CIFAR-10 | validation accuracy at epoch 50: 49.9% -> 69.3% | Helped |
| Increase K, the number of adapted sharpest directions (ablation) | NSGD · CIFAR-10 | training speed | Helped |
| Increase K, the number of adapted sharpest directions (ablation) | NSGD · CIFAR-10 | maximum spectral norm: approximately 8·10^3 compared to approximately 150 | Mixed |
| Use γ<1 along adapted directions (ablation) | NSGD · CIFAR-10 | test accuracy: 86.4% -> 87.0% | Helped |
| Use γ>1 along adapted directions (ablation) | NSGD · CIFAR-10 | test accuracy: 86.4% -> 85.5% | Hurt |
| Use γ<1 along adapted directions (ablation) | NSGD · IMDB | test accuracy: 88.79% -> 88.83% | No clear effect |
| Increase the SGD learning rate (ablation) | SGD · CIFAR-10 and PTB | maximum spectral norm along training | Hurt |
| Decrease the SGD batch size (ablation) | SGD · CIFAR-10 and PTB | maximum spectral norm along training | Hurt |
| Increase momentum (ablation) | SGD · VGG-11 on CIFAR-10 and CIFAR-100 | maximum spectral norm along training | Hurt |
| Use only the top-eigenvector gradient projection (ablation) | SimpleCNN · CIFAR-10 | loss | Hurt |
| Remove the gradient component along the top eigenvector (ablation) | SimpleCNN · CIFAR-10 | sharpness | Hurt |
Closest papers by meaning
- Does SGD really happen in tiny subspaces? (ICLR 2025 · similarity 0.90)
- Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and More (ICML 2025 · similarity 0.89)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
- A Precise Characterization of SGD Stability Using Loss Surface Geometry (ICLR 2024 · similarity 0.89)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.88)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.