Research map / Architectures and efficiency / Quantization
M+Adam: Low-Precision Training via Additive–Multiplicative Optimization
Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup, Anima Anandkumar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multiplicative update branch (M+Adam instead of AdamW) | AdamW · C4/EN 1x Chinchilla BF16/BF16, 350M | validation perplexity: 19.156 -> 16.607 | Improved |
| add multiplicative update branch (M+Adam instead of AdamW) | AdamW · C4/EN 1x Chinchilla BF16/BF16, 60M | validation perplexity: 29.865 -> 29.035 | Improved |
| add multiplicative branch instead of AdamW | AdamW · C4/EN 1x Chinchilla NVFP4/FP8, 350M | validation perplexity: 24.941 -> 20.796 | Improved |
| add multiplicative branch instead of AdamW | AdamW · C4/EN 1x Chinchilla NVFP4/FP8, 60M | validation perplexity: 33.884 -> 30.948 | Improved |
| add multiplicative branch instead of AdamW | AdamW · C4/EN 1x Chinchilla FP8/FP8, 350M | validation perplexity: 21.674 -> 19.395 | Improved |
| add multiplicative branch instead of AdamW | AdamW · C4/EN 1x Chinchilla BF16/FP8, 1B | validation perplexity: 16.346 -> 15.690 | Improved |
| use M+Adam instead of AdamW with stochastic rounding | AdamW + SR · C4/EN 1x Chinchilla BF16/BF16, 350M | validation perplexity: 18.720 -> 16.607 | Improved |
| use M+Adam instead of AdamW with Kahan compensation | AdamW + Kahan · C4/EN 1x Chinchilla BF16/BF16, 130M | validation perplexity: 22.859 -> 22.032 | Improved |
| use stochastic rounding with AdamW | AdamW · C4/EN 1x Chinchilla BF16/BF16, 1B | validation perplexity: 15.642 -> 14.346 | Improved |
| compress multiplicative second-moment state with Apollo-E rank 4 (ablation) | M+Adam · C4/EN 1x Chinchilla BF16/BF16, 350M | validation perplexity: 16.607 -> 16.550 | No clear effect |
| increase Apollo-E rank from 4 to 16 (ablation) | M+Adam with Apollo-E · C4/EN 1x Chinchilla BF16/BF16, 130M | validation perplexity: 21.979 -> 21.995 | No clear effect |
| use mini tensor-wise Apollo-E instead of channel-wise rank 1 (ablation) | M+Adam with Apollo-E · C4/EN 1x Chinchilla BF16/BF16, 350M | validation perplexity: 16.683 -> 16.752 | Hurt |
Closest papers by meaning
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.86)
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.86)
- MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence (NeurIPS 2024 · similarity 0.85)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.85)
- MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · similarity 0.85)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
