Research map / Optimization / Implicit bias and generalization

On the Convergence Analysis of Muon

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Explicit and implicit regularization interplay · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Gives convergence rate analysis of Muon (SVD-based orthogonalized momentum updates) under Frobenius, spectral, and non-uniform smoothness assumptions, and compares it with GD to show when Muon can win by exploiting low-rank Hessian structure.

Problem

Muon's convergence behavior and why it outperforms GD-based optimizers on matrix-structured neural network parameters are not theoretically understood; classical GD analysis depends on the maximum Hessian eigenvalue, which can be large or vary during training.

Approach

Analyzes Muon under Frobenius-norm smoothness, spectral-norm smoothness, and a new averaged-Hessian quantity J (under bounded third derivative), showing its complexity can depend on the average of Hessian terms along the trajectory rather than the maximum singular value, which can be much smaller when Hessians are low-rank or blockwise diagonal (Kronecker-structured).

Key results

• On MNIST linear regression with 100 output classes, the ratio D_F^2 L/(D_op^2 L_*) reaches 3.52 and Muon more clearly outperforms GD and Adam; on Gaussian features (ratio 224.06 for ||X||_F^2/||X||_op^2) Muon performs worse than GD.
• Theory: under Frobenius smoothness Muon's complexity O(r^2 L sigma^2 Delta eps^-4) and O(rL Delta eps^-2) is no better than (S)GD's, while spectral smoothness gives O(r L_* sigma^2 Delta eps^-4) and O(L_* Delta eps^-2).
• In a six-layer GPT-2 style model, the empirical ratio ||grad||_*^2/J_t exceeds ||grad||_F^2/L_t for Muon versus GD/AdamW across tested steps (reported qualitatively).

Surprising findings

• Under the conventional Frobenius-norm Lipschitz smoothness assumption, Muon's convergence rate is no better than and may be worse than (S)GD.
• For Gaussian random feature matrices, Muon underperforms GD, indicating advantage depends on data low-rankness.

Limitations

• Analysis is for SVD-based Muon in full theory; practical Newton-Schulz approximation is not analyzed.
• The advantage relies on assumptions (low-rank Hessian, high-rank W0-W*, bounded third derivative) that are only partially verified empirically; J_t comparisons are empirical for multi-sample nonconvex settings.
• Theoretical guarantees are stated up to constants and some bounds rely on assumed bounded iterate distance D_op.

Experimental setup

Datasets: MNIST Shakespeare character-level corpus CIFAR-10

Benchmarks: MNIST Shakespeare character-level corpus

Metrics: loss gradient nuclear norm gradient Frobenius norm ratio of J_t to L_t

Baselines: GD GD with Nesterov momentum Adam AdamW SGD

Builds on: Muon Lion/steepest descent under spectral norm Frobenius-smoothness analysis of SGD LMO-based non-Euclidean trust-region methods

Closest papers by meaning

• Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.89)
• From SGD to Muon: Adaptive Optimization via Schatten-p Norms (NeurIPS 2026 · similarity 0.88)
• Muon Does Not Converge on Convex Lipschitz Functions (NeurIPS 2026 · similarity 0.88)
• AMUSE: Anytime Muon with Stable Gradient Evaluation (NeurIPS 2026 · similarity 0.88)
• Tensorion: A Tensor-Aware Generalization of the Muon Optimizer (NeurIPS 2026 · similarity 0.88)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.87)