Research map / Optimization / Implicit bias and generalization
Exploring Landscapes for Better Minima along Valleys
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALTO, an adaptor "E" that adds an EMA-of-EMA acceleration term of gradient differences to gradient-based optimizers so they keep exploring valleys for flatter, lower minima, and applies it to Lamb for large-batch training.
Problem
Gradient-based optimizers stop once they reach a local minimum, so they cannot guarantee the lowest or best-generalizing minimum; large-batch training is especially hard, with poor generalization at large batch sizes.
Approach
Replace gradient g_k with g_k + alpha*a_k, where a_k is an EMA of successive gradient differences (approximating the negative gradient of squared gradient norm, i.e., repulsion from sharp minima); with alpha<0 the optimizer explores along valleys and tends toward flatter minima. Adapter applied to SGD, Adam, and Lamb (yielding ESGD, EAdam, ALTO).
Key results
- ALTO achieves 70.83% top-1 on ImageNet with batch size 4086 versus SGD's 70.64% at batch size 256 (ResNet34).
- On GPT-2 (345M) pretraining with batch size 4096, ALTO reaches test perplexity 78.37 versus Lamb's 83.13.
- ALTO outperforms Lamb across ResNet50 ImageNet batch sizes 1K-32K (e.g., 77.22 vs 77.06 at 1K; 76.70 vs 76.42 at 32K).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ALTO instead of SGD | ResNet20 · CIFAR10, batch 128 | top-1 accuracy (%): 91.85 -> 91.24 | Hurt |
| use ALTO instead of AdaBelief | ResNet20 · CIFAR10, batch 16384 | top-1 accuracy (%): 88.03 -> 88.83 | Helped |
| use ALTO instead of SGD | ResNet20 · CIFAR100, batch 128 | top-1 accuracy (%): 64.93 -> 65.74 | Helped |
| use ALTO instead of Lamb | ResNet20 · CIFAR100, batch 16384 | top-1 accuracy (%): 56.06 -> 57.78 | Helped |
| use ALTO instead of SGD | ResNet34 · ImageNet, batch 256 | top-1 accuracy (%): 70.64 -> 69.95 | Hurt |
| use ALTO instead of Lamb | ResNet34 · ImageNet, batch 4086 | top-1 accuracy (%): 70.34 -> 70.83 | Helped |
| use ALTO instead of Lamb | ResNet50 · ImageNet, batch 1K–32K | top-1 accuracy (%): 77.06$^\dagger$ -> 77.22; 77.11$^\dagger$ -> 77.25; 76.92$^\dagger$ -> 77.35; 76.89$^\dagger$ -> 77.10; 76.66$^\dagger$ | Helped |
| use ALTO instead of Lamb | GPT-2 (345M) · GPT-2 Output Dataset (1M), batch 4096 | test perplexity: 83.13 -> 78.37 | Helped |
| add acceleration term a_m to m (ablation) | ALTO · CIFAR10, batch 16384 | top-1 accuracy (%): 87.06 -> 88.83 | Helped |
| add acceleration term a_v to v (ablation) | ALTO · CIFAR10, batch 16384 | top-1 accuracy (%): 87.08 -> 88.83 | Helped |
| add bias correction (ablation) | ALTO · CIFAR10, batch 16384 | top-1 accuracy (%): 87.51 -> 88.83 | Helped |
| add layerwise learning-rate regularization (ablation) | ALTO · CIFAR10, batch 16384 | top-1 accuracy (%): 86.22 -> 88.83 | Helped |
Surprising findings
- Adding the acceleration term increases GPU memory only about 2% over Lamb at large batch sizes, with the relative overhead independent of model size.
- Negative alpha (exploration) is preferred for large batches, while positive alpha is used for small batches, reversing the usual intuition about acceleration.
Limitations
- Two extra hyperparameters (beta1 and alpha) whose optimal values vary across tasks and require tuning.
- Improvements are limited in the small-batch regime.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet CoNLL-2003 IMDB MRPC GPT-2 Output Dataset
Benchmarks: CIFAR-10 CIFAR-100 ImageNet MRPC GPT-2 Output Dataset
Metrics: top-1 accuracy test perplexity F1 training time to target accuracy GPU memory
Baselines: SGD Adam AdamW Lamb AdaBelief Lion Adabelief
Builds on: Lamb Adam AdamW Lion Adan
Closest papers by meaning
- Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers (ICML 2021 · similarity 0.88)
- Learned Optimizers that Scale and Generalize (ICML 2017 · similarity 0.88)
- Asymmetric Valleys: Beyond Sharp and Flat Local Minima (NeurIPS 2019 · similarity 0.88)
- Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes (CVPR 2025 · similarity 0.87)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.87)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.