Research map / Optimization / Optimizers
Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond
Pan Zhou, Hanshu YAN, Xiaotong Yuan, Jiashi Feng, Shuicheng YAN
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add local regularizer beta_q/2||theta-theta_{q-1}||^2 to stagewise lookahead (SLRLA vs SLA) | SLA · CIFAR10 ResNet18 | accuracy (%): 95.27 -> 95.47 | Improved |
| add local regularizer to stagewise lookahead (SLRLA vs SLA) | SLA · CIFAR100 VGG16 | accuracy (%): 70.20 -> 70.63 | Improved |
| add local regularizer to stagewise lookahead (SLRLA vs SLA) | SLA · ImageNet ResNet18 | accuracy (%): 70.30 -> 70.47 | Improved |
| add local regularizer to stagewise lookahead (SLRLA vs SLA) | SLA · CIFAR10 WRN-16-10 | accuracy (%): 95.73 -> 96.08 | Improved |
| add lookahead (SLA) on top of stagewise SGD | Stagewise SGD · CIFAR100 WRN-16-10 | accuracy (%): 78.95 -> 79.54 | Improved |
| use SLRLA instead of adaptive optimizer Adam | Adam · CIFAR100 ResNet18 | accuracy (%): 76.88 -> 78.58 | Improved |
| use SLRLA instead of adaptive optimizer AdamW | AdamW · CIFAR10 VGG16 | accuracy (%): 90.75 -> 92.63 | Improved |
| use SLRLA instead of AdaBelief | AdaBelief · ImageNet ResNet18 | accuracy (%): 70.08 -> 70.47 | Improved |
| increase lookahead alpha | lookahead and SLA · regularized softmax MNIST | training loss and test loss | Mixed |
| increase alpha for SLRLA | SLRLA · regularized softmax MNIST | test accuracy | Mixed |
| tune regularization constant beta_q over a large range | SLRLA · ImageNet | accuracy | No clear effect |
| use stagewise LR decay with lookahead rather than linear decay (theory) | lookahead · regularized softmax MNIST | excess risk | Improved |
Closest papers by meaning
- Improving Sharpness-Aware Minimization by Lookahead (ICML 2024 · similarity 0.84)
- Lookbehind-SAM: k steps back, 1 step forward (ICML 2024 · similarity 0.84)
- EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization (NeurIPS 2026 · similarity 0.84)
- Stability and Generalization of Asynchronous SGD: Sharper Bounds Beyond Lipschitz and Smoothness (NeurIPS 2024 · similarity 0.84)
- Investigating the Role of Weight Decay in Enhancing Nonconvex SGD (CVPR 2025 · similarity 0.83)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
