Research map / Optimization / Nonconvex and smooth optimization
Stochastic Anderson Mixing for Nonconvex Stochastic Optimization
Fuchao Wei, Chenglong Bao, Yang Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AdaSAM instead of the strongest listed baseline | ResNet18, ResNet20, ResNet32, ResNet44, ResNet56, and WideResNet16-4 · CIFAR-10 | test top-1 accuracy (%): 94.82±.15 -> 95.17±.10; 92.07±.04 -> 92.43±.19; 92.86±.15 -> 93.22±.32; 93.26±.24 -> 93.57±.14; 93.76±.16 -> 93.77±.12; | Improved |
| Use AdaSAM instead of the strongest listed baseline | ResNet18, ResNeXt50, and DenseNet121 · CIFAR-100 | test top-1 accuracy (%): 77.63±.35 -> 78.13±.14; 78.93±.12 -> 79.31±.27; 79.37±.16 -> 80.09±.52 | Improved |
| Add adaptive regularization and damped projection rather than use RAM | Anderson mixing on CIFAR-10 and CIFAR-100 · CIFAR-10 and CIFAR-100 | test top-1 accuracy (%): 95.10±.05 -> 95.17±.10; 92.21±.09 -> 92.43±.19; 93.05±.43 -> 93.22±.32; 93.42±.13 -> 93.57±.14; 93.76±.16 -> 93.77±.12; | Improved |
| Add variance reduction using SVRG (ablation) | AdaSAM · MNIST training | training loss: 10^{-9} | Improved |
| Precondition AdaSAM with Adagrad or RMSprop (ablation) | AdaSAM · MNIST training | training loss: 10^{-3} | Mixed |
| Precondition AdaSAM with Adam (ablation) | AdaSAM on Penn TreeBank LSTM · Penn TreeBank | perplexity | Improved |
| Alternate vanilla SGD with AdaSAM every 5 steps (ablation) | AdaSAM · CIFAR-100, ResNeXt50 | test top-1 accuracy (%): 78.41±.54 -> 79.47±.44 | Improved |
| Alternate Adam with AdaSAM every 5 steps (ablation) | Adam · CIFAR-10 and CIFAR-100 | test top-1 accuracy (%) | Mixed |
| Use alternating optimizer steps to reduce computational overhead (ablation) | AdaSAM-SGD and AdaSAM-Adam · CIFAR-10 and CIFAR-100 | test top-1 accuracy (%): 79.31±.27 -> 79.47±.44 | Mixed |
| Use vanilla AdaSAM without a suitable preconditioner (ablation) | AdaSAM · Penn TreeBank LSTM | perplexity | Hurt |
| Use preconditioned mixing (ablation) | AdaSAM · MNIST training | training loss: 10^{-3} | Mixed |
Closest papers by meaning
- A Class of Short-term Recurrence Anderson Mixing Methods and Their Applications (ICLR 2022 · similarity 0.92)
- A Variant of Anderson Mixing with Minimal Memory Size (NeurIPS 2022 · similarity 0.90)
- Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilization (NeurIPS 2021 · similarity 0.88)
- ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimization (JMLR 2020 · similarity 0.87)
- Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization (ICML 2025 · similarity 0.85)
- Faster Double Adaptive Gradient Methods (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
