Research map / Optimization / Nonconvex and smooth optimization
Muon Does Not Converge on Convex Lipschitz Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Muon (and signed momentum on diagonal functions) fails to converge on convex Lipschitz objectives for any learning-rate schedule, proposes EF-Muon with error feedback that restores anytime convergence, but finds it hurts practical performance on CIFAR-10 and nanoGPT.
Problem
Whether Muon converges on the class of convex Lipschitz (nonsmooth) functions, a class that has guided AdaGrad, Shampoo, and Schedule-Free design, and whether convex Lipschitz theory is a useful model for Muon.
Approach
On diagonal matrices Muon reduces exactly to signed momentum, which allows adapting a signed subgradient counterexample to show Muon non-convergence for any momentum beta; error feedback applied to LMO-based steps (viewing the sharp operator as a compression operator) restores convergence, but this fix does not help practical training.
Key results
- On CIFAR-10 MuonMax reaches validation CE 0.265 vs vanilla Muon 0.287 and EF-MuonMax 0.388.
- On nanoGPT (206M, FineWeb-Edu 10B) vanilla Muon reaches 3.01 validation CE, MuonMax 3.27, EF-MuonMax 3.43.
- Counterexamples show Muon iterates stay at f(W_t)-inf f >= 1-beta for an open set of initializations under offline nonincreasing stepsizes with positive limit, and fail to converge for almost every initialization when beta in [0,1/2).
Surprising findings
- Error feedback, which provably restores convergence for Muon on convex Lipschitz functions, degrades practical performance on both CIFAR-10 and nanoGPT.
- Muon on diagonal matrices is exactly signed momentum, linking Muon to Adam-style sign methods.
Limitations
- Offline decreasing stepsize counterexample excludes regularized Muon; the adaptive counterexample only covers beta in [0,1/2).
- Neither counterexample considers a momentum scheduler; complexity results are not ruled out.
- Approximate polar factor computation and Nesterov momentum are not analyzed.
Experimental setup
Datasets: CIFAR-10 FineWeb-Edu 10B
Benchmarks: CIFAR-10 FineWeb-Edu 10B
Metrics: validation cross-entropy loss
Baselines: Vanilla Muon MuonMax EF-MuonMax
Builds on: Muon signed momentum error feedback (EF-signSGD) AdaGrad AMSGrad
Closest papers by meaning
- On the Convergence Analysis of Muon (NeurIPS 2026 · similarity 0.88)
- AMUSE: Anytime Muon with Stable Gradient Evaluation (NeurIPS 2026 · similarity 0.87)
- Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.87)
- Some Fundamental Aspects about Lipschitz Continuity of Neural Networks (ICLR 2024 · similarity 0.86)
- Phases of Muon: When Muon Eclipses SignSGD (NeurIPS 2026 · similarity 0.85)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.