Research map / Optimization / Optimizers
Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization
Zitao Song, Cedar Site Bai, Zhe Zhang, Brian Bullins, David F. Gleich
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DeVA_S_inf instead of Muon | NanoGPT 270M · modded-NanoGPT FineWeb-Edu | validation loss: 3.305 -> 3.271 | Improved |
| use DeVA_S_inf instead of SOAP | NanoGPT 270M · modded-NanoGPT FineWeb-Edu | validation loss: 3.304 -> 3.271 | Improved |
| use DeVA_S_inf instead of Muon | NanoGPT 270M · modded-NanoGPT FineWeb-Edu | training time: 10.72s -> 16.49s | Hurt |
| use SOAP instead of Muon | NanoGPT 270M · modded-NanoGPT FineWeb-Edu | validation loss: 3.305 -> 3.304 | No clear effect |
| raise lr from 1e-3 to 1e-2 | Muon · modded-NanoGPT LR sweep | validation loss: 3.3038 -> 3.4015 | Hurt |
| raise lr from 1e-3 to 1e-2 | DeVA_S_inf · modded-NanoGPT LR sweep | validation loss: 3.2709 -> 3.4174 | Hurt |
| use DeVA_S_inf instead of Muon at lr 1e-3 | optimizer · modded-NanoGPT LR sweep | validation loss: 3.3038 -> 3.2709 | Improved |
| use DeVA_S_inf instead of SOAP at lr 1e-4 | optimizer · modded-NanoGPT LR sweep | validation loss: 3.3686 -> 3.3450 | Improved |
| use DeVA_S_inf instead of Muon at lr 1e-4 | optimizer · modded-NanoGPT LR sweep | validation loss: 3.2819 -> 3.3450 | Hurt |
Closest papers by meaning
- MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · similarity 0.87)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.87)
- Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad (NeurIPS 2024 · similarity 0.87)
- Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (NeurIPS 2025 · similarity 0.86)
- Decoupling Angles and Strength in Low-rank Adaptation (ICLR 2025 · similarity 0.86)
- Free Decompression with Algebraic Spectral Curves (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
