Research map / Optimization / Implicit bias and generalization
Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay
Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, Jian Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SGDM (momentum 0.9) instead of plain SGD at same learning rate and weight decay | ResNet50 on ImageNet, fixed learning rate · ImageNet | time to reach equilibrium | Improved |
| add momentum (larger alpha) at fixed learning rate and weight decay | ResNet50 on ImageNet · ImageNet | training/test loss | Hurt |
| vary learning rate and weight decay while keeping their product (angular update) the same (ablation) | ResNet50 SGD on ImageNet · ImageNet | training loss and test accuracy | No clear effect |
| increase angular update in equilibrium via larger learning rate or weight decay (ablation) | ResNet50 on ImageNet · ImageNet | training loss and test error | Hurt |
| rescale scale-invariant weight norm by 10^(1/4) whenever learning rate is divided by 10 | SGDM multi-stage schedule, ResNet50 · ImageNet | angular update vs theoretical value | Improved |
| rescale weight norm at learning rate decay | Mask-RCNN ResNet50-FPN 4x schedule · MSCOCO | angular update vs theoretical value | Improved |
| rescale weight norm at learning rate decay to remove increasing angular update | ResNet18 SGDM on CIFAR10 · CIFAR10 | test accuracy drop in 2nd learning rate stage | Improved |
| use Adam instead of SGDM | ResNet18 on CIFAR10 · CIFAR10 | test accuracy drop after learning rate decay | Improved |
Closest papers by meaning
- Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup (NeurIPS 2019 · similarity 0.87)
- Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics (ICLR 2021 · similarity 0.86)
- Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes (NeurIPS 2022 · similarity 0.86)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.86)
- Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws (NeurIPS 2025 · similarity 0.85)
- Does SGD really happen in tiny subspaces? (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
