Research map / Optimization / Optimizers
MoMo: Momentum Models for Adaptive Learning Rates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoMo, a Polyak-type adaptive learning rate built on a momentum-averaged, truncated model of the loss that wraps SGD-M and Adam (MoMo-Adam) and an online lower-bound estimator.
Problem
Standard momentum methods such as SGD-M and Adam require extensive learning-rate tuning, which is computationally expensive for training large neural networks.
Approach
Build a model of the full loss by exponentially averaging past sampled losses and their linearizations (momentum as a model), truncate it at a lower bound, and take a proximal step; this yields a closed-form adaptive learning rate that generalizes the stochastic Polyak step size to momentum and preconditioned methods.
Key results
- MoMo and MoMo-Adam keep good performance over a learning-rate range extended by over one order of magnitude compared to SGD-M and Adam on CIFAR/Criteo tasks.
- MoMo-Adam* on IWSLT14 transformer is on par or better than tuned AdamW across base learning rates and both dropout values.
- Convergence O(1/sqrt(K)) for convex problems under interpolation without knowing problem constants other than the optimal value.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the MoMo adaptive learning rate to SGD-M | SGD-M · CIFAR100, CIFAR10, Criteo, and MNIST | validation accuracy | Helped |
| Add the MoMo adaptive learning rate to Adam | Adam · CIFAR100, CIFAR10, Criteo, and MNIST | validation accuracy | Helped |
| Use the adaptive learning rate | MoMo and MoMoAdam · CIFAR100, CIFAR10, Criteo, and MNIST | validation accuracy | Mixed |
| Add the MoMoAdam adaptive learning rate | AdamW · ImageNet-1k | top-1 validation accuracy | Helped |
| Use lower bound 0 instead of an informative lower bound (ablation) | MoMo and MoMoAdam · ImageNet32 | validation accuracy | No clear effect |
| Use oracle lower bound 0.9 (ablation) | MoMo and MoMoAdam · ImageNet32 | validation accuracy | Helped |
| Estimate the loss lower bound online (ablation) | MoMo and MoMoAdam · ImageNet32 | validation accuracy | Helped |
| Add MoMoAdam* with online lower-bound estimation | AdamW · IWSLT14 German-to-English, dropout 0.1 and 0.3 | BLEU | Mixed |
| Use MoMoAdam instead of AdamW | ViT and diffusion model training with warmup and cosine decay · ImageNet-1k and Smithsonian Butterflies | validation accuracy | Helped |
Surprising findings
- Without momentum, MoMo reduces to the stochastic Polyak step size SPS_max.
- With zero lower bound, MoMo's adaptive term often gives little improvement on tasks where the final loss is far from zero, requiring the online lower-bound estimate.
Limitations
- Convergence theory restricted to convex losses under interpolation.
- Lower-bound estimation requires bootstrapping and is heuristic for non-convex losses.
- Comparisons for some tasks limited to MoMo-Adam vs AdamW due to compute cost.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 ImageNet-1k ImageNet32 Criteo IWSLT14 Smithsonian Butterflies
Benchmarks: MNIST CIFAR10 CIFAR100 ImageNet-1k ImageNet32 Criteo IWSLT14
Metrics: validation accuracy training loss BLEU sensitivity to learning rate
Baselines: SGD-M Adam AdamW SPS_max Adabelief Adabound Lion
Builds on: SGD with momentum Adam AdamW stochastic Polyak step size D-adaptation
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.86)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.86)
- ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-Order Optimization (AAAI 2024 · similarity 0.85)
- Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees (ICML 2025 · similarity 0.85)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.85)
- SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.