Research map / Optimization / Implicit bias and generalization
Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train ResNet101 on ImageNet with the standard staged learning-rate schedule | ResNet101 · ImageNet | gradient norm convergence | Hurt |
| Use a constant learning rate instead of the standard staged schedule | ResNet101 · ImageNet | gradient norm convergence | No clear effect |
| Reduce the constant learning rate | ResNet101 · ImageNet | gradient norm | Hurt |
| Continue training for 300 epochs at step size 0.01 | ResNet101 · ImageNet | gradient norm convergence | No clear effect |
| Train Transformer-XL on WT103 with cosine or constant learning-rate schedules | Transformer-XL · WT103 | gradient norm convergence | Hurt |
| Train a ResNet on the smaller Cifar10 task | ResNet · Cifar10 | gradient norm convergence | Helped |
| Use gradient descent with learning rate 0.01 rather than 0.04 | Gradient descent on the synthetic objective · Synthetic objective | gradient norm convergence: 0.01 vs 0.04 | Helped |
Closest papers by meaning
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.88)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.88)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.88)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.