Research map / Optimization / Optimizers
Delving into Muon and Beyond: Deep Analysis and Extensions
Xianbiao Qi, Marco Chen, Jiaquan Ye, Yelin He, Rong Xiao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Muon orthogonalization (p=0) to momentum instead of raw momentum SGD | mSGD · OpenWebText val loss | validation loss: 3.092 -> 2.887 | Improved |
| use moderate spectral compression p=1/4 instead of full orthogonalization on momentum (ablation) | mSGDZ (Muon) · OpenWebText val loss | validation loss: 2.887 -> 2.876 | Improved |
| use p=1/2 instead of p=1 on momentum (ablation) | mSGD · OpenWebText val loss | validation loss: 3.092 -> 2.895 | Improved |
| add RMS normalization before the update instead of momentum input (ablation) | Muon (p=0) · OpenWebText val loss | validation loss: 2.887 -> 2.879 | Improved |
| add full orthogonalization (p=0) to RMS-normalized input (ablation) | Adam · OpenWebText val loss | validation loss: 2.871 -> 2.879 | Hurt |
| add mild compression p=1/2 to RMS-normalized input (ablation) | Adam · OpenWebText val loss | validation loss: 2.871 -> 2.870 | No clear effect |
| add p=1/4 compression to RMS-normalized input (ablation) | Adam · OpenWebText val loss | validation loss: 2.871 -> 2.872 | No clear effect |
| raise learning rate from 1e-1 to 10.0 beyond the optimum (ablation) | mSGD · OpenWebText val loss | validation loss: 3.649 -> 3.164 | Mixed |
| use Muon orthogonalization instead of mSGD for lr robustness (ablation) | momentum optimizers · OpenWebText val loss | validation loss: 3.649 -> 2.891 | Improved |
| add spectral compression p=1/2 to RMS-normalized input at small lr (ablation) | Adam · OpenWebText val loss | validation loss: 2.884 -> 2.918 | Mixed |
Closest papers by meaning
- Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.91)
- Phases of Muon: When Muon Eclipses SignSGD (NeurIPS 2026 · similarity 0.91)
- Tensorion: A Tensor-Aware Generalization of the Muon Optimizer (NeurIPS 2026 · similarity 0.90)
- From SGD to Muon: Adaptive Optimization via Schatten-p Norms (NeurIPS 2026 · similarity 0.90)
- Muon Dynamics as a Spectral Wasserstein Flow (NeurIPS 2026 · similarity 0.89)
- Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
