Research map / Optimization / Implicit bias and generalization
Rethinking Neural Network Learning Rates: A Stackelberg Perspective
Sihan Zeng, Sujay Bhatt, Sumitra Ganesh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a larger learning rate for the final layer than the body (two-time-scale updates) | two-layer ReLU network, synthetic regression with SGD · Synthetic regression (objective f(M_k, w_k)) | objective value | Improved |
| evaluate body update on the Stackelberg objective instead of the joint objective | two-layer ReLU network, early training · Synthetic regression landscape | largest Hessian eigenvalue: 2.8 -> 21.9 (iteration 5) | Improved |
| curvature advantage late in training | Stackelberg objective · Synthetic regression landscape | largest Hessian eigenvalue: 19.4 vs 17.9 (iteration 200) | No clear effect |
| non-uniform learning rates instead of uniform small (alpha) (ablation) | regression, classification, Atari DQN with RMSProp · Friedman, Housing, MNIST, Fashion, CIFAR10, Atari | training/performance curves | Improved |
| scale all layers up to the larger learning rate (uniform large) instead of non-uniform (ablation) | regression, classification, Atari DQN with RMSProp · Friedman, Housing, MNIST, Fashion, CIFAR10, Atari | stability / final performance | Hurt |
| non-uniform learning rates versus uniform rates on some problems | neural network training · general | performance | No clear effect |
Closest papers by meaning
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.86)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.85)
- Where Do Large Learning Rates Lead Us? (NeurIPS 2024 · similarity 0.85)
- The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training (ICML 2025 · similarity 0.85)
- Accelerating neural network training: An analysis of the AlgoPerf competition (ICLR 2025 · similarity 0.85)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
