Research map / Optimization / Optimizers

Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering

NeurIPS 2026 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that momentum acts as a spectral filter inside Muon, and that applying momentum before the polar (orthogonalization) step gives provably better signal alignment than reversing the order or removing momentum.

Problem

The theoretical role of momentum inside Muon is unclear: prior analyses either drop momentum or retain it without explaining why it helps, and standard theory suggests heavy momentum does not help. The paper also asks why momentum should precede orthogonalization.

Approach

Model gradients as a coherent low-rank signal plus bounded-variance mean-zero orthogonal perturbation. Momentum averages out the perturbation, opening a spectral gap that grows with the effective window T=1/(1-beta), which stabilizes singular subspaces; because the polar factor flattens all singular values, orthogonalizing before momentum (Post-polar) or without momentum (Polar-only) erases this b

Key results

• Momentum-buffer tail singular values decay as (2T-1)^{-1/4}, and the top-r singular subspace error decreases at rate O(T^{-1/4}) (theory, with probability at least 1-(2T-1)^{-1/2}).
• In the rank-1 spiked Gaussian model under low SNR, Polar-only signal alignment is bounded by 1/sqrt(m) plus terms vanishing as m grows, while Pre-polar still recovers the signal for large T.
• On the stationary NanoGPT probe, Pre-polar full-rank signal alignment rises monotonically with beta, while Post-polar and Polar-only stay low and nearly flat.

Surprising findings

• Orthogonalizing each per-step gradient before momentum averaging (Post-polar, Orthogonal-SGDM style) yields almost no signal alignment, nearly as poor as Polar-only.
• Standard optimization theory predicts degradation under heavy momentum, but the spectral-gap analysis predicts improvement as beta approaches 1.

Limitations

• Rank-1 spiked Gaussian model is used for the quantitative separation result; extension to general rank-r signals and non-Gaussian perturbations is left open.
• Assumes signal persistence and time-invariant signal for the main ordering theorem; the schedule-dependent (increasing momentum) case is not analyzed.
• Nesterov momentum as used in practice is not covered by the analysis.

Experimental setup

Datasets: CIFAR-10

Benchmarks: CIFAR-10 NanoGPT validation loss LLaMA 350M validation loss

Metrics: validation loss signal alignment (full-rank and rank-5) subspace alignment error sin-theta spectral filtering ratio noise-suppression ratio

Baselines: Muon Post-polar (Orthogonal-SGDM) Polar-only AdamW

Builds on: Muon Orthogonal-SGDM Wedin sin-theta theorem heavy-ball momentum

Closest papers by meaning

• Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.88)
• Muon Dynamics as a Spectral Wasserstein Flow (NeurIPS 2026 · similarity 0.86)
• Phases of Muon: When Muon Eclipses SignSGD (NeurIPS 2026 · similarity 0.86)
• AMUSE: Anytime Muon with Stable Gradient Evaluation (NeurIPS 2026 · similarity 0.85)
• Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients? (NeurIPS 2026 · similarity 0.85)
• Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization (NeurIPS 2026 · similarity 0.85)