Research map / Optimization / Optimizers

Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective

NeurIPS 2026 · Optimization › Optimizers · approach: Adam convergence and adaptive learning rates · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Develops a mixed-spiked matrix sensing testbed and river-valley analysis showing Muon is faster early but oscillates near the optimum versus GD, and proposes a Muon-to-AdamW two-stage schedule validated on a 250M LLaMA-style pretraining run.

Problem

Muon's empirical gains over Adam/AdamW in LLM training are mixed and schedule-sensitive, and existing theory mostly highlights its advantages without characterizing its limitations along the full optimization trajectory.

Approach

Introduces a mixed-spiked matrix sensing model (signal, spike, bulk components) and a river-valley decomposition of the reduced dynamics, showing Muon's constant-magnitude sign-like updates speed early river progress but overshoot near the optimum, motivating a switch to GD-like refinement.

Key results

• In LLM pretraining (250M params, 4k iterations), Muon-to-AdamW hybrids with peak LR 1e-3 and 1.5e-3 reach lower final validation loss than all Muon-only LR-schedule baselines.
• Theory shows GD needs O((1/eta) log(1/eta)) post-transition iterations to refine the river variable while fixed-step Muon needs O(1/eta) and oscillates near the equilibrium.
• Constant-LR Muon exhibits the fastest initial loss decrease before the switch in LLM training.

Surprising findings

• Muon's admissible step size shrinks proportionally to the residual scale, so Muon-only training needs decaying schedules, whereas GD does not.

Limitations

• Theory is based on a simplified momentum-free Muon and a specific mixed-spiked matrix sensing model; generalization is informal.
• LLM experiments are preliminary, small-scale (250M, 4k iterations) and single-setting.
• Relies on assumptions such as gradient invariance near the target and momentum aligned with the river.

Experimental setup

Datasets: OpenWebText2 synthetic mixed-spiked matrix sensing instances

Benchmarks: OpenWebText2 validation loss

Metrics: training loss validation loss

Baselines: Muon with cosine LR Muon with linear LR Muon with cos_inf LR AdamW

Builds on: river-valley perspective (WSD schedules) Muon AdamW Burer-Monteiro factorization

Closest papers by meaning

• AMUSE: Anytime Muon with Stable Gradient Evaluation (NeurIPS 2026 · similarity 0.92)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.90)
• On the Convergence Analysis of Muon (NeurIPS 2026 · similarity 0.89)
• From SGD to Muon: Adaptive Optimization via Schatten-p Norms (NeurIPS 2026 · similarity 0.89)
• MuLoCo: Muon is a Practical Inner Optimizer for DiLoCo (ICML 2026 · similarity 0.89)
• Phases of Muon: When Muon Eclipses SignSGD (NeurIPS 2026 · similarity 0.88)