Research map / Optimization / Implicit bias and generalization
On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regime
Shuai Jiang, Eric C Cyr, Ben S. Southworth, Alexey Voronin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LM/Gauss-Newton preconditioning instead of SGD | MLP regression on sum-of-sines target · 1D regression, mode-wise FFT error | convergence across frequencies | Improved |
| use LM preconditioning instead of SGD | PINN Poisson solver · Poisson equation with (1,1),(2,2),(3,3) forcing | training loss at higher frequency | Improved |
| use LM preconditioning instead of SGD/AdamW | MNIST grokking via init scale alpha · MNIST grokking | generalization delay | Improved |
| use LM preconditioning instead of AdamW | MNIST grokking via init scale alpha · MNIST grokking | final test accuracy | Hurt |
| use LM preconditioning instead of SGD | modular addition MLP with output scaling alpha · modular addition grokking | train-test delay | Improved |
| use LM preconditioning instead of SGD | high-dimensional polynomial regression · polynomial regression grokking | train-test loss delay | Improved |
| use generalized Gauss-Newton instead of Adam | transformer on modular addition · modular addition (transformer) | final validation accuracy: 100% -> 45% | Hurt |
| switch from LM to AdamW after 2000 iterations | MNIST grokking · MNIST grokking | final test accuracy | Improved |
Closest papers by meaning
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.91)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.90)
- Grokking at the Edge of Linear Separability (ICML 2025 · similarity 0.90)
- Grokking Beyond the Euclidean Norm of Model Parameters (ICML 2025 · similarity 0.89)
- Why Do You Grok? A Theoretical Analysis on Grokking Modular Addition (ICML 2024 · similarity 0.88)
- Deep Networks Always Grok and Here is Why (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
