Research map / Optimization / Implicit bias and generalization
How Does Adaptive Optimization Impact Local Neural Network Geometry?
Kaiqi Jiang, Dhruv Malik, Yuanzhi Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Adam instead of SGD+M | BERT-small IMDB · iter 750 layer 22 | R_med: 233.72 -> 72.67 | Improved |
| Train with Adam instead of SGD+M | BERT-small IMDB · iter 1250 layer 9 | R_med: 11.43 -> 14.24 | Hurt |
| Train with Adam instead of SGD+M | Transformer on Multi30k · epoch 30 layer 15 | R_med: 52.41 -> 7.5 | Improved |
| Train with Adam instead of SGD+M | Transformer on Multi30k · epoch 30 layer 12 | R_med: 6.64 -> 7.25 | No clear effect |
| Train on random targets instead of real targets (Adam vs SGD+M gap) | Transformer · epoch 30 layer 12 | R_med: 133.39 -> 3.91 | Improved |
| Add weight decay 0.001 to both optimizers | Adam vs SGD+M · Multi30k epoch 55 layer 24 | R_med: 53.79 -> 3.32 | Improved |
| Add weight decay 0.001 to both optimizers | Adam vs SGD+M · Multi30k epoch 30 layer 7 | R_med: 8.22 -> 39.65 | Hurt |
| Use large weight decay | Adam · translation | R_med | Hurt |
| Switch task to image classification (ResNet on CIFAR-10) | Adam vs SGD+M · CIFAR-10 epoch 20 layer 1 | R_med: 3.74 -> 39.35 | Hurt |
| Switch from SGD to Adam midway | Adam after SGD · Multi30k layer 14 | R_med: 14.34 -> 5.33 | Improved |
| Switch from SGD to Adam midway | Adam after SGD · Multi30k layer 13 | R_med: 294.76 -> 332.96 | No clear effect |
| Compare Adam and SGD+M at equal training loss | Adam · BERT-small and Multi30k | R_med | Improved |
Closest papers by meaning
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.88)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.88)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.87)
- Convergence of Distributed Adaptive Optimization with Local Updates (ICLR 2025 · similarity 0.87)
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.87)
- The Optimization Landscape of SGD Across the Feature Learning Strength (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
