Research map / Optimization / Optimizers
Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes signSGD on high-dimensional linear regression via a homogenized sign SGD process and deterministic ODEs for the risk, deriving exact risk curves that quantify preconditioning and noise-compression effects.
Problem
Understanding the dynamics of signSGD (and by extension Adam-like sign-based optimizers) before convergence is hard to quantify in theoretically solvable settings; existing work mostly addresses long-term convergence.
Approach
Study one-pass signSGD on linear regression in a high-dimensional limit, showing its dynamics concentrate around a homogenized SDE and a risk ODE (for Gaussian data and noise), with an arcsin-based drift in the covariance-preconditioned direction; compare against vanilla SGD via learning-rate optimization and compensation.</👍core_idea>
Key results
- For isotropic Gaussian data and noise, the homogenized risk ODE for signSGD is dR/dt = -4 eta R/(pi sqrt(2R+eta^2)) + eta^2/2.
- With learning-rate optimization in the isotropic Gaussian case, signSGD is strictly worse than SGD by a factor (2/pi)^2.
- Under constant learning rate in the scaled isotropic case, signSGD converges to a nonzero risk when noise is present; its limit goes to 0 as alpha grows, so signSGD can beat SGD only for large enough alpha.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SignSGD instead of SGD with the learning rate optimized for each method | SignSGD vs. SGD · Gaussian data and noise | risk descent rate: (2/\pi)^2 | Hurt |
| compensate SignSGD's learning rate to match SGD's mean term | SignSGD · isotropic data with Gaussian noise | risk descent rate: (\pi/2)^2 | Hurt |
| use SignSGD instead of SGD with a fixed scaled-isotropy learning rate | SignSGD vs. SGD · scaled isotropy, K = αI, Gaussian data and noise | long-term risk: R_\infty = \frac{\eta \sigma^2}{2 ( 2- \eta)}; R_\infty = \frac{\eta^2 \pi^2 / \alpha + \eta / \alpha \sqrt{\eta^2 \pi^2 | Mixed |
| use SignSGD instead of SGD with a fixed learning rate | SignSGD vs. SGD · scaled isotropy, noiseless case (σ = 0) | long-term risk: R_\infty = \eta^2 \pi^2 \alpha / 32 | Hurt |
Surprising findings
- With Gaussian data and noise, signSGD is worse than SGD even after optimizing the learning rate, contrary to the intuition that sign preconditioning helps.
Limitations
- Results are derived mainly for Gaussian data and Gaussian noise; general noise distributions are only heuristic.
- Proofs contain many unresolved notes and placeholders in the source.
- Momentum is not considered.
Experimental setup
Datasets: synthetic Gaussian linear regression data
Metrics: risk (population excess loss) curves
Baselines: vanilla SGD
Builds on: signSGD Adam SGD DMFT-style ODE comparison
Closest papers by meaning
- When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds (NeurIPS 2026 · similarity 0.91)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.90)
- Phases of Muon: When Muon Eclipses SignSGD (NeurIPS 2026 · similarity 0.90)
- On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent (ICLR 2025 · similarity 0.90)
- Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise (NeurIPS 2026 · similarity 0.87)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.