Research map / Optimization / Implicit bias and generalization
Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
Yuqing Wang, Minshuo Chen, Tuo Zhao, Molei Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a sufficiently large constant learning rate instead of an infinitesimal or diminishing learning rate | Gradient Descent on homogeneous matrix factorization · General matrix factorization | factor imbalance | Improved |
| Use large learning rate on an initially unbalanced scalar factorization | Gradient Descent · Scalar factorization, μ=1, ||x₀||=18, ||y₀||=0.09 | factor difference: 17.9986 to 0.0154 | Improved |
| Use larger learning rates within the convergence range | Gradient Descent on scalar factorization · Scalar factorization | factor norm gap | Improved |
| Use small learning rates rather than large learning rates | Gradient Descent on scalar factorization · Scalar factorization | factor norm gap | No clear effect |
| Use learning rates above 2/L | Gradient Descent on scalar factorization · Scalar factorization | loss monotonicity | Hurt |
| Use learning rates above 2/L within the analyzed range | Gradient Descent on scalar factorization · Scalar factorization | convergence to a global minimum | Improved |
| Use learning rates that are too large | Gradient Descent on matrix factorization · General matrix factorization | convergence | Hurt |
| Use larger learning rates near the stability limit | Gradient Descent · Matrix sensing | factor norm balance | Improved |
| Use larger learning rates | Gradient Descent · Matrix completion | factor norm gap | Improved |
| Use step size h=1.9 | Gradient Descent on (1-xy)²/2 · Scalar factorization | convergence to a fixed global minimum | Hurt |
Closest papers by meaning
- Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced (NeurIPS 2018 · similarity 0.90)
- Flatland: The Adventures of Gradient Descent with Large Step Sizes (ICML 2026 · similarity 0.89)
- Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult (JMLR 2025 · similarity 0.89)
- Train faster, generalize better: Stability of stochastic gradient descent (ICML 2016 · similarity 0.88)
- Convergence Rates for Gradient Descent on the Edge of Stability for Overparametrised Least Squares (NeurIPS 2025 · similarity 0.87)
- Provable Acceleration of Nesterov's Accelerated Gradient for Asymmetric Matrix Factorization and Linear Neural Networks (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
