Research map / Optimization / Optimizers
Lookbehind-SAM: k steps back, 1 step forward
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lookbehind, a SAM wrapper that takes multiple ascent steps to find higher-loss worst-case perturbations and then linearly interpolates slow and fast weights to stabilize the descent step, improving generalization, robustness to weight noise, and lifelong learning.
Problem
SAM uses a single ascent step to approximate the inner maximization, which can be inaccurate; naively taking multiple ascent steps or increasing the neighborhood size rho hurts training loss and generalization due to gradient instability and drift from the original point.
Approach
Perform k SAM-style ascent steps per iteration (looking behind by climbing the loss) with fast weights, then update slow weights by linear interpolation toward the fast weights, so gradients from intermediate distances are combined to reduce variance in the descent step; an adaptive alpha based on gradient alignment removes one hyperparameter.
Key results
- Lookbehind-SAM reaches 96.27% on CIFAR-10 with ResNet-34 vs 95.80% for SAM, and Lookbehind-ASAM reaches 96.54% vs 96.32% for ASAM.
- Lookbehind-SAM is the only method that outperforms vanilla SAM and ASAM on ImageNet (ResNet-18: 70.16% vs 70.01% for SAM).
- On Split-CIFAR100 with ER, ER + Lookbehind-SAM reaches 68.87% average accuracy and 12.37 forgetting vs 68.28% and 13.98 for ER + SAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SAM with Lookbehind-SAM | SAM · CIFAR-10 (ResNet-34) | validation accuracy (%): 95.80_{\pm .07} -> 96.27_{\pm .07} | Helped |
| Replace ASAM with Lookbehind-ASAM | ASAM · CIFAR-100 (WRN-28-10) | validation accuracy (%): 81.67_{\pm .28} -> 82.16_{\pm .09} | Helped |
| Replace SAM with Lookbehind-SAM | SAM · ImageNet (ResNet-18) | validation accuracy (%): 70.01_{\pm .06} -> 70.16_{\pm .08} | Helped |
| Add adaptive α to Lookbehind-SAM (ablation) | Lookbehind-SAM · CIFAR-10 (ResNet-34) | validation accuracy (%): 96.27_{\pm .07} -> 96.33_{\pm .04} | Helped |
| Add adaptive α to Lookbehind-SAM (ablation) | Lookbehind-SAM · CIFAR-100 (ResNet-50) | validation accuracy (%): 78.62_{\pm .48} -> 78.33_{\pm .36} | Hurt |
| Add adaptive α to Lookbehind-ASAM (ablation) | Lookbehind-ASAM · CIFAR-10 (ResNet-34) | validation accuracy (%): 96.54_{\pm .21} -> 96.57_{\pm .03} | Helped |
| Add adaptive α to Lookbehind-ASAM (ablation) | Lookbehind-ASAM · ImageNet (ViT-Base) | validation accuracy (%): 81.87 -> 81.89 | Mixed |
| Add multiple ascent steps and aggregate/interpolate the updates (Lookbehind vs. Multistep-SAM) | SAM · CIFAR-100 (WRN-28-10) | validation accuracy (%): 80.55_{\pm .06} -> 80.99_{\pm .02} | Helped |
| Use gradient averaging in Multistep-SAM (ablation) | Multistep-SAM · CIFAR-10 (ResNet-34) | validation accuracy (%): 95.72_{\pm .15} -> 95.74_{\pm .25} | No clear effect |
| Use adaptive α with Lookbehind-$m$-SAM at α=adaptive, k=2 (ablation) | m-SAM · CIFAR-10 (ResNet-34), m=64 | validation accuracy (%): 95.99_{\pm .04} -> 96.49_{\pm .05} | Helped |
| Use Lookbehind-$m$-SAM with α=0.2, k=2 (ablation) | m-SAM · CIFAR-10 (ResNet-34), m=64 | validation accuracy (%): 95.99_{\pm .04} -> 95.91_{\pm .08} | Hurt |
| Increase ascent steps k (ablation) | Lookbehind-SAM · CIFAR-10 and CIFAR-100 | validation accuracy (%) | Helped |
Surprising findings
- Multistep SAM variants without interpolation give little or no improvement over single-step SAM, and increasing rho in SAM degrades performance.
- Going farther from the original point with rho*k outperforms staying within rho/k, so the default rho need not be re-tuned.
Limitations
- Adds two hyperparameters (k and alpha) on top of rho and eta; adaptive alpha still adds one.
- Computational overhead increases training time by roughly a factor k.
- Lifelong learning meta-learning evaluation only uses the constant learning rate C-MAML variant.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet Split-CIFAR100 Split-TinyImageNet IWSLT'14 (DE-EN)
Benchmarks: CIFAR-10 CIFAR-100 ImageNet Split-CIFAR100 Split-TinyImageNet IWSLT'14 (DE-EN)
Metrics: validation accuracy m-sharpness accuracy under multiplicative weight noise average accuracy forgetting test BLEU
Baselines: SGD SAM ASAM Multistep-SAM Lookahead-SAM Lookahead-SGD ER Lookahead-C-MAML
Builds on: SAM ASAM Lookahead optimizer Lookahead-MAML m-SAM
Closest papers by meaning
- Improving Sharpness-Aware Minimization by Lookahead (ICML 2024 · similarity 0.91)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.90)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.90)
- Zeroth-Order Sharpness-Aware Learning with Exponential Tilting (NeurIPS 2026 · similarity 0.90)
- Tilted Sharpness-Aware Minimization (ICML 2025 · similarity 0.89)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.