Research map / Optimization / Optimizers

AMUSE: Anytime Muon with Stable Gradient Evaluation

NeurIPS 2026 · Optimization › Optimizers · approach: Adam convergence and adaptive learning rates · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AMUSE, a Schedule-Free variant of Muon that evaluates gradients at a time-varying interpolation between fast and averaged iterates, removing learning-rate schedules and improving the performance-iteration frontier over AdamW and Muon.

Problem

Muon's orthogonalized updates speed progress along the river-valley loss landscape but amplify valley-wall oscillations, and standard recipes need explicit learning-rate schedules that hinder anytime training.

Approach

Combine Muon's orthogonalized matrix updates with Schedule-Free interpolation, using a time-varying coefficient beta_t that starts near the fast sequence and shifts toward the averaged sequence so gradients are evaluated at points with smaller dominant-subspace components.

Key results

• AMUSE reaches Muon's final performance with 1.51x fewer steps on 720M Llama pretraining, 1.34x fewer on ImageNet ResNet-50, and 2.57x fewer on ImageNet ViT fine-tuning.
• AMUSE achieves lower validation perplexity than all baselines throughout training across Llama 124M, 720M, and 1.3B scales.
• AMUSE requires no learning rate decay schedule and supports anytime training, with no extra memory beyond one additional state copy versus vanilla Muon.

Surprising findings

• Muon's orthogonalization sharply reduces dominant ratio relative to raw momentum yet still amplifies small residual dominant components, causing bouncing across valley walls.
• SF-AdamW struggles with large batch sizes in LLM pretraining while AMUSE scales effectively to 720M and 1.3B models.

Limitations

• Requires one extra state copy (z_t) relative to vanilla Muon, adding memory overhead.
• Only one additional hyperparameter rho plus beta_init, which were tuned per setting; Muon momentum fixed at 0.95.

Experimental setup

Datasets: FineWeb-100B CIFAR-10 CIFAR-100 SVHN ImageNet-1k ISIC 2018 MNIST SST-2

Benchmarks: CIFAR-10 CIFAR-100 SVHN ImageNet-1k ISIC 2018 FineWeb validation perplexity

Metrics: validation perplexity accuracy dominant component ratio steps to reach baseline performance

Baselines: AdamW Schedule-Free AdamW Muon SGD SF-SGD SF-Muon

Builds on: Schedule-Free optimizer Muon river-valley landscape Newton-Schulz orthogonalization AdamW

Closest papers by meaning

• Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.92)
• From SGD to Muon: Adaptive Optimization via Schatten-p Norms (NeurIPS 2026 · similarity 0.90)
• On the Convergence Analysis of Muon (NeurIPS 2026 · similarity 0.88)
• Tensorion: A Tensor-Aware Generalization of the Muon Optimizer (NeurIPS 2026 · similarity 0.88)
• Muon Does Not Converge on Convex Lipschitz Functions (NeurIPS 2026 · similarity 0.87)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.86)