Research map / Optimization / Optimizers
General Analysis of LMO-based Optimizers: Beyond Bounded Variance
Egor Shulgin, Mohamed Awad, Peter Richtárik, Eduard Gorbunov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase momentum alpha (ablation) | Muon (idealized) on matrix regression · Matrix regression, eps = 10^-1 | empirical critical batch size: bcrit = 1 -> bcrit = 16 (alpha 0.1 -> 0.9) | Improved |
| increase momentum alpha with larger batch size (optimal momentum grows with batch) (ablation) | normalized SGD with momentum · Linear regression, eps = 10^-2 | optimal batch size: b* = 1 for alpha <= 0.5, larger for higher alpha | Improved |
| use full-batch updates for a harder target accuracy (ablation) | normalized SGD with momentum · Linear regression, eps = 5e-3 | sample complexity | Improved |
| increase batch size under bounded-variance theory | LMO-family methods · theory | sample complexity | Hurt |
| increase momentum from 0.1 to 0.2 (ablation) | Muon/AdamW mix on NanoGPT (~6M params) · WikiText-103, validation loss < 3.9 | epochs to target loss | Mixed |
| increase batch size (ablation) | Muon/AdamW mix on NanoGPT · WikiText-103 | optimal step size | Improved |
Closest papers by meaning
- L-SVRG and L-Katyusha with Arbitrary Sampling (JMLR 2021 · similarity 0.88)
- From SGD to Muon: Adaptive Optimization via Schatten-p Norms (NeurIPS 2026 · similarity 0.87)
- Stochastic Momentum Methods for Non-smooth Non-Convex Finite-Sum Coupled Compositional Optimization (NeurIPS 2025 · similarity 0.86)
- Phases of Muon: When Muon Eclipses SignSGD (NeurIPS 2026 · similarity 0.86)
- Stochastic Zeroth-Order Optimization Under Heavy-Tailed Noise (NeurIPS 2026 · similarity 0.86)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
