Research map / Optimization / Optimizers
Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives the first formal SDE for SignSGD and new SDEs for AdamW and RMSpropW under weaker assumptions, analyzes noise effects (three phases, linear vs quadratic noise dependence, resilience to heavy-tailed noise), and validates via Euler-Maruyama integration on MLP, CNN, ResNet, and Transformer.
Problem
Adaptive optimizers (SignSGD, RMSprop, Adam, AdamW) are widely used but their theoretical understanding, especially the precise role of gradient noise in their dynamics and asymptotic behavior, is incomplete; prior SDE models for them are restrictive or unvalidated.
Approach
Derive weak-approximation SDEs for SignSGD, AdamW and RMSpropW under general noise assumptions (including heavy-tailed Student's t), showing SignSGD's loss and iterate covariance scale linearly with noise (vs quadratically for SGD), and that decoupled weight decay bounds the asymptotic loss of AdamW as noise grows.
Key results
- SignSGD's asymptotic loss scales linearly in noise level sigma while SGD's scales quadratically (Lemma on strongly convex loss and quadratic stationary distribution).
- For AdamW, the asymptotic loss is upper-bounded in noise level, whereas Adam with L2 regularization has asymptotic loss growing linearly in sigma.
- The SDE of SignSGD under Student's t noise with nu=2 has drift constant sqrt(1/2) vs sqrt(2/pi) for Gaussian noise, differing by ~10%, so SignSGD remains resilient where SGD diverges.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase gradient-noise level | SignSGD · Strongly convex objectives, Phase 3 | convergence speed | Hurt |
| Increase gradient-noise level | SGD · Strongly convex objectives | convergence speed | No clear effect |
| Increase gradient-noise level | SignSGD · Strongly convex objectives, asymptotic | expected loss | Hurt |
| Increase gradient-noise level | SGD · Strongly convex objectives, asymptotic | expected loss | Hurt |
| Use SignSGD instead of SGD under high gradient noise | SignSGD and SGD · Strongly convex objectives, asymptotic | expected loss | Mixed |
| Use infinite-variance Student's t gradient noise instead of Gaussian noise | SignSGD · SignSGD dynamics, Phase 3 | drift coefficient: ∼ 10% apart | No clear effect |
| Use infinite-variance gradient noise | SignSGD versus SGD · Heavy-tailed noise | performance | Helped |
| Add decoupled weight decay | AdamW · Convex quadratic, high-noise asymptotic | expected loss | Helped |
| Use coupled L2 regularization instead of decoupled weight decay (ablation) | Adam · Convex quadratic, high-noise asymptotic | expected loss | Hurt |
| Rescale weight decay θ according to the scaling rule (ablation) | AdamW · Convex quadratic, batch-size scaling | asymptotic expected loss | Helped |
| Vary β₁ and β₂ (ablation) | AdamW · Non-convex landscape | asymptotic loss level | No clear effect |
| Vary β₁ and β₂ (ablation) | AdamW · Non-convex landscape | basin selection and convergence speed | Mixed |
Surprising findings
- Decoupled weight decay in AdamW makes the asymptotic loss bounded even as noise variance goes to infinity, unlike Adam with L2 regularization.
- SignSGD's convergence speed is inversely affected by noise, unlike SGD where noise does not affect convergence speed.
Limitations
- SDE approximations rely on regularity assumptions (smooth densities, polynomial growth) that may fail for large step sizes
- Stationary-distribution and scaling-rule results are derived mainly for quadratic or strongly convex losses
- Validation of Euler-Maruyama integration is computationally expensive since full gradients are needed each step
Experimental setup
Datasets: Breast Cancer MNIST CIFAR-10 SlimPajama
Benchmarks: Breast Cancer MNIST CIFAR-10 SlimPajama
Metrics: loss f(x) trajectories asymptotic loss iterate covariance
Baselines: SGD Adam RMSprop SignSGD previous SDEs for Adam and RMSprop
Builds on: Li et al. SDE framework for stochastic optimizers SignSGD AdamW RMSprop
Closest papers by meaning
- Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects (ICML 2025 · similarity 0.90)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.89)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.89)
- On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent (ICLR 2025 · similarity 0.88)
- Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise (NeurIPS 2026 · similarity 0.87)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.