Research map / Architectures and efficiency / Quantization
A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization
Xuan Tang, Jichu Li, Difan Zou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reduce mantissa length M (very low) for quantized training (ablation) | Adam and Muon · Rosenbrock, CIFAR-10 MLP, nanoGPT | convergence / gradient norm | Hurt |
| increase mantissa length to moderate values (ablation) | Adam and Muon · Rosenbrock, CIFAR-10 MLP, nanoGPT | convergence vs full precision | Improved |
| increase mantissa bit-length (ablation) | Adam and Muon · Rosenbrock (10,000 iterations) | converged gradient norm | Improved |
| push beta2 toward 1 under quantization (ablation) | Adam · synthetic and real-data experiments | sensitivity to quantization error | Hurt |
| use Muon instead of AdamW under low-precision training | nanoGPT on OpenWebText · nanoGPT (26M parameters) low-precision training | training robustness | Improved |
Closest papers by meaning
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.89)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.89)
- MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence (NeurIPS 2024 · similarity 0.88)
- ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-Order Optimization (AAAI 2024 · similarity 0.88)
- On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm (NeurIPS 2025 · similarity 0.88)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
