Research map / Optimization / Optimizers
Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Removes the square root from adaptive methods (RMSProp, full/diag AdaGrad, Shampoo) and gives a second-order (Gaussian variational / natural-gradient) derivation, yielding root-free and inverse-free matrix methods that work in half precision.
Problem
Adaptive gradient methods like Adam(W) and Shampoo apply a square root to their gradient-outer-product preconditioner, which conflates sign descent with adaptivity, breaks the second-order motivation, and makes matrix methods numerically expensive in low precision. The paper asks how adaptive method
Approach
Removing the root makes the preconditioner an inverse-covariance update; the gradient outer product is reinterpreted as a new empirical Fisher over a joint label distribution (for mini-batch-averaged losses, with a scaling fix), which gives scale and affine invariance. Via Bayesian learning rule/natural gradient on a Gaussian variational approximation, this yields root-free updates and a "precondi
Key results
- Root-free RMSProp closes the generalization gap to SGD on CNNs (CIFAR-100, ResNet50) while matching root-based RMSProp/AdamW on vision transformers (ImageWoof10).
- IF-Shampoo runs in BFP-16 where root-based Shampoo needs single or double precision, and one step takes less than half the time of Shampoo while using less memory.
- Root-free methods with random search (200 runs) and step-decay schedules perform well on SwinViT, FocalNet, GCViT and VMamba, with matrix methods outperforming diagonal ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove the square root from RMSProp | RMSProp · CIFAR-100 | generalization gap to SGD | Helped |
| Remove the square root from RMSProp | RMSProp · ImageWoof10 | performance | No clear effect |
| Replace Shampoo with inverse- and root-free IF-Shampoo | Shampoo · GCViT, SwinViT, FocalNet, and VMamba | per-iteration progress | No clear effect |
| Replace Shampoo with inverse- and root-free IF-Shampoo | Shampoo · GCViT, SwinViT, FocalNet, and VMamba | training speed: twice as fast | Helped |
| Remove the square root from adaptive methods in the original training setup | Adaptive methods · Original training setup | performance | Hurt |
Surprising findings
- Removing the square root closes the generalization gap between adaptive methods and SGD on CNNs, contrary to the sign-descent explanation.
- Under the original constant-learning-rate training scheme, root-based methods outperform root-free ones, so the root helps in the outdated setting.
Limitations
- Root-free methods underperform root-based ones under the original constant learning rate schedule and without tuning.
- Non-zero preconditioner initialization and extra learning rate beta2 are needed for stability; PD correction needed for arbitrary curvature approximations.
Experimental setup
Datasets: CIFAR-100 ImageWoof10
Benchmarks: CIFAR-100 ImageWoof10
Metrics: test error generalization gap wall-clock time memory
Baselines: AdamW RMSProp SGD Shampoo diag-AdaGrad full-AdaGrad
Builds on: Adam RMSProp AdaGrad Shampoo Bayesian learning rule
Closest papers by meaning
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.86)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.85)
- AdaFisher: Adaptive Second Order Optimization via Fisher Information (ICLR 2025 · similarity 0.85)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.85)
- Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad (NeurIPS 2024 · similarity 0.85)
- Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.