Research map / Optimization / Implicit bias and generalization

Phases of Muon: When Muon Eclipses SignSGD

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Gradient flow and mean-field dynamics · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives exact deterministic risk dynamics for stochastic spectral optimizers (SignSVD/Muon) and SignSGD on a high-dimensional matrix least-squares problem, showing batch-dependent spectral resolution and three phases in the power-law (alpha, beta) plane.

Problem

Understanding when and why spectral optimizers such as Muon outperform Adam-like methods, and how their training dynamics depend on batch size, dimension, and data covariance structure.

Approach

Uses a random matrix theory deterministic-equivalent analysis of a matrix-valued linear regression to obtain a closed scalar/projected risk recursion with drift and volatility kernels for SignSVD and SignSGD, revealing a batch-dependent resolution threshold and a phase diagram under power-law covariance.

Key results

• SignSVD on isotropic data has optimal learning rate exceeding SignSGD's by sqrt(N) for B>=N and by N/sqrt(B) for B<=N, but identical convergence rate up to constants at optimal learning rate.
• With power-law covariance, the (alpha, beta) plane splits into three phases (SignSVD favored, SignSGD favored, trade-off), with boundaries at beta=1 and beta=alpha+1.
• Batch size acts as a spectral resolution scale: SignSVD whitens the top i_# modes (i_# ~ B for alpha>1) and behaves like SGD on the remaining modes for B<N.

Surprising findings

• At optimal learning rates on isotropic data, SignSVD gives no convergence speedup over SignSGD despite the larger learning rate.
• SignSGD on a Haar-rotated covariance fails to capture square-root preconditioning, whereas SignSVD is basis-independent.

Limitations

• Analysis restricted to linear least squares with no label noise, no feature learning, and no edge-of-stability effects.
• No momentum in main analysis; momentum treated only numerically in the appendix.
• Proportional high-dimensional limit; results on the Muon/Newton-Schulz approximation validated numerically rather than analytically.

Experimental setup

Datasets: synthetic Gaussian matrix least-squares data

Metrics: population risk time to 4-epsilon approximate solution convergence rate optimal learning rate

Baselines: SignSGD Adam (via SignSGD proxy) SGD

Builds on: Muon SignSGD Adam deterministic equivalents for SGD and SignSGD

Closest papers by meaning

• When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds (NeurIPS 2026 · similarity 0.90)
• Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects (ICML 2025 · similarity 0.90)
• Muon Dynamics as a Spectral Wasserstein Flow (NeurIPS 2026 · similarity 0.88)
• Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.88)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.88)
• Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives (NeurIPS 2026 · similarity 0.87)