Research map / Optimization / Optimizers
A Momentumized, Adaptive, Dual Averaged Gradient Method
Aaron Defazio, Samy Jelassi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use stage-wise learning rate schedule instead of sqrt-decay a/sqrt(i+b) for SGD (ablation) | SGD without momentum · CIFAR-10 | test accuracy (%): 84.25 -> 94.15 | Improved |
| increase offset b in sqrt-decay schedule from 1 to 10000 (ablation) | SGD without momentum · CIFAR-10 | test accuracy (%): 84.25 -> 93.55 | Improved |
| add adaptivity (AdaGrad denominator) to dual averaging under stage-wise schedule (ablation) | Dual averaging (DA) · CIFAR-10 | test accuracy (%): 84.48 -> 93.66 | Improved |
| use stage-wise instead of flat learning rate for AdaGrad-DA (ablation) | AdaGrad-DA · CIFAR-10 | test accuracy (%): 91.85 -> 93.66 | Improved |
| use stage-wise instead of flat learning rate for plain dual averaging (ablation) | Dual averaging (DA) · CIFAR-10 | test accuracy (%): 87.03 -> 84.48 | Hurt |
| use dual averaging AdaGrad instead of mirror descent AdaGrad (stage-wise) (ablation) | AdaGrad · CIFAR-10 | test accuracy (%): 92.05 -> 93.66 | Improved |
| MADGRAD instead of Adam | PreResNet152 · CIFAR-10 | test accuracy (%): 94.69 -> 95.38 | Improved |
| MADGRAD instead of SGD+M | ResNet-50 · ILSVRC 2012 ImageNet | test accuracy (%): 76.09 -> 76.22 | Improved |
| MADGRAD instead of Adam | ResNet-50 · ILSVRC 2012 ImageNet | test accuracy (%): 73.03 -> 76.22 | Improved |
| MADGRAD instead of Adam | VarNet 2.0 · fastMRI Knee | test SSIM: 0.9111 -> 0.9119 | Improved |
| MADGRAD instead of Adam | LSTM · IWSLT14 De-En | test loss: 4.35 -> 4.33 | Improved |
| MADGRAD instead of Adam | RoBERTa · BookWiki masked LM | test loss: 2.09 -> 2.07 | Improved |
Closest papers by meaning
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.91)
- Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad (NeurIPS 2024 · similarity 0.88)
- AdaGrad under Anisotropic Smoothness (ICLR 2025 · similarity 0.87)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.87)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.87)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
