Research map / Optimization / Optimizers

From SGD to Muon: Adaptive Optimization via Schatten-p Norms

NeurIPS 2026 · Optimization › Optimizers · approach: Second-order and structured preconditioning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SMuon, which adaptively picks a layer-wise Schatten-p LMO geometry (interpolating SGDM and Muon) via a closed-form single-step random-feature proxy, combined with Adam-style preconditioning, and an efficient Taylor/Horner fractional polar factor approximation.

Problem

Existing LMO-based optimizers (SGDM, Muon, MuAdam) fix the update geometry by design or heuristics, which may not be optimal for each layer's problem geometry during training.

Approach

Derive a closed-form layer-wise proxy-optimal Schatten exponent p* from gradient, momentum and activation spectral statistics under a single-step random feature regression model, and use the resulting Schatten-(p+1) LMO update with p-dependent Adam-style second-order preconditioning, computed via a Taylor expansion built on Newton-Schulz polar factor estimates.

Key results

• On Modded-NanoGPT across 6 seeds, SMuon (Adam) reaches 3.366 ± 3.1e-3 validation loss vs Muon's 3.376 ± 4.3e-3, improving on every seed.
• On low-rank LoRA fine-tuning of Qwen-2.5-0.5B on GSM8K, AdamW reaches 0.337 train loss, SMuon (Adam) 0.340, Muon 0.387, SGDM 0.736.
• SMuon (Adam) runtime overhead is about 3% on NanoGPT (2 A100s) and about 0.6% on vision (RTX 4090).

Surprising findings

• On LoRA fine-tuning, SMuon without second-order preconditioning performs worse than SGDM-like baselines in validation accuracy (28.18 vs 22.38 for SGDM is actually higher, but train loss 0.464 is worse than Muon).
• Adaptive p* gives a modest gain over a fixed p=50 Muon-like geometry, and the reported gains are small in absolute terms.

Limitations

• Gains over Muon are modest in absolute terms and the proxy is derived from a single-step random feature regression surrogate that ignores downstream Jacobians.
• Element-wise preconditioned p* criterion is not scalable and not used in experiments; the optimal step size variant is also omitted due to cost and loss spikes.
• Vision experiments run only 20 epochs, too short to reach AdamW performance.

Experimental setup

Datasets: FineWeb ImageNette GSM8K MNIST

Benchmarks: FineWeb validation loss (NanoGPT speedrun) ImageNette validation accuracy GSM8K

Metrics: validation loss train loss validation accuracy

Baselines: AdamW Muon SGDM MuAdam SMuon

Builds on: Muon LMO theory (Pethick et al.) MuAdam Newton-Schulz iteration random feature regression spectral descent analysis

Closest papers by meaning

• Tensorion: A Tensor-Aware Generalization of the Muon Optimizer (NeurIPS 2026 · similarity 0.90)
• AMUSE: Anytime Muon with Stable Gradient Evaluation (NeurIPS 2026 · similarity 0.90)
• Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.89)
• On the Convergence Analysis of Muon (NeurIPS 2026 · similarity 0.88)
• MuLoCo: Muon is a Practical Inner Optimizer for DiLoCo (ICML 2026 · similarity 0.87)
• Phases of Muon: When Muon Eclipses SignSGD (NeurIPS 2026 · similarity 0.87)