Research map / Optimization / Nonconvex and smooth optimization
Faster Double Adaptive Gradient Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 2AdaSGD and 2AdaSPIDER, nonconvex stochastic optimizers that jointly adapt learning rate (Adam-like diagonal preconditioner) and mini-batch size (via stochastic gradient mappings), with SFO complexity bounds for nonsmooth regularization.
Problem
Nonconvex finite-sum optimization with possibly nonsmooth convex regularization, where existing adaptive gradient methods adapt either learning rate or batch size but high-complexity joint methods (AdAdaGrad) exist.
Approach
Adapt the mini-batch size per iteration using the average squared norm of recent stochastic gradient mappings, combined with an adaptive proximal preconditioned update; a SPIDER variance-reduced variant (2AdaSPIDER) achieves optimal SFO complexity under average smoothness.
Key results
- 2AdaSGD achieves SFO complexity O(nε^-1 ∧ ε^-2) and 2AdaSPIDER achieves O(√n ε^-1 ∧ ε^-3/2), matching the known lower bound for finding ε-stationary points in nonconvex finite-sum optimization.
- In experiments on CIFAR-10, ImageNet, PTB and WikiText2 with and without L1 regularization, the proposed methods are reported to basically outperform the compared adaptive methods.
- The adaptive batch size is set from average squared norm of stochastic gradient mappings, with minimum batch size capped by sigma^2 eps^-1 and n.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 2AdaSGD with adaptive learning rate and adaptive mini-batch size | 2AdaSGD · CIFAR-10 image classification without regularization | classification performance | Helped |
| Use 2AdaSPIDER with adaptive learning rate and adaptive mini-batch size | 2AdaSPIDER · CIFAR-10 image classification with L1 regularization | classification performance | Helped |
| Use 2AdaSGD with adaptive learning rate and adaptive mini-batch size | 2AdaSGD · ImageNet image classification without regularization | classification performance | Helped |
| Use 2AdaSPIDER with adaptive learning rate and adaptive mini-batch size | 2AdaSPIDER · ImageNet image classification with L1 regularization | classification performance | Helped |
| Use 2AdaSGD with adaptive learning rate and adaptive mini-batch size | 2AdaSGD · Penn Treebank language modeling without regularization | language-modeling performance | Helped |
| Use 2AdaSPIDER with adaptive learning rate and adaptive mini-batch size | 2AdaSPIDER · Penn Treebank language modeling with L1 regularization | language-modeling performance | Helped |
| Use 2AdaSGD with adaptive learning rate and adaptive mini-batch size | 2AdaSGD · WikiText2 language modeling without regularization | language-modeling performance | Helped |
| Use 2AdaSPIDER with adaptive learning rate and adaptive mini-batch size | 2AdaSPIDER · WikiText2 language modeling with L1 regularization | language-modeling performance | Helped |
Limitations
- Experiments are limited to small models (3-layer CNN, 2-layer LSTM/Transformer) and ResNet18 on ImageNet.
- Worst-case complexity bounds depend on assumptions such as bounded variance and L-smoothness of components (average smoothness for 2AdaSPIDER).
Experimental setup
Datasets: CIFAR-10 ImageNet Penn Treebank WikiText2
Benchmarks: CIFAR-10 ImageNet Penn Treebank WikiText2
Metrics: training loss vs. epochs/iterations gradient norm
Baselines: Adam Adam-type (AMSGrad) Adagrad AdaBelief Super-Adam AbaSPIDER AdaSPIDER PROXGEN AdAdaGrad
Builds on: Adam SPIDER AdAdaGrad PROXGEN AdaSPIDER
Closest papers by meaning
- Proximal Stochastic Methods for Nonsmooth Nonconvex Finite-Sum Optimization (NeurIPS 2016 · similarity 0.88)
- Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization (NeurIPS 2024 · similarity 0.87)
- Faster Stochastic Optimization with Arbitrary Delays via Adaptive Asynchronous Mini-Batching (ICML 2025 · similarity 0.87)
- Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness (ICLR 2025 · similarity 0.87)
- Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction (NeurIPS 2024 · similarity 0.86)
- Adaptive Algorithms with Sharp Convergence Rates for Stochastic Hierarchical Optimization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.