Research map / Continual learning / Online and replay-based
Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a perturbed-forgetting view of SAM dynamics and an output bias forgetting (OBF) perturbation that pushes predictions toward uniformity during the ascent step, improving ImageNet and robustness generalization over SAM, GSAM, and ASAM.
Problem
SAM's sharpness metric does not reliably correlate with generalization, and the mechanisms behind SAM's benefits (including small perturbation batch advantages) are not well explained by flatness.
Approach
Interpret SAM perturbations as transient forgetting of model biases (via an information bottleneck argument on output-exposed biases), and design the OBF perturbation that ascends a loss pushing predictions toward a uniform distribution, with a per-sample alpha based on the target likelihood, instead of steepest ascent on the task loss.
Key results
- On ImageNet with ViT-S/32, SAM with OBF reaches 74.49% top-1 on ImageNet-V1 versus 72.77% for steepest-ascent SAM, and 25.31% vs 21.63% on ImageNet-R.
- With ViT-S/16, SAM+OBF achieves 80.30% ImageNet-V1 vs 78.73% for standard SAM.
- Transfer: ViT-S/32 pretrained with OBF-SAM reaches 97.79% on CIFAR-10 and 87.04% on CIFAR-100 vs 97.74% and 86.94% for steepest-ascent SAM pretraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace steepest-ascent perturbations with OBF perturbations | SAM on ViT-S/32 · ImageNet benchmarks and robustness datasets | top-1 accuracy: 72.77 -> 74.49; 78.89 -> 81.31; 58.81 -> 61.13; 21.63 -> 25.31; 19.68 -> 22.58 | Helped |
| Replace steepest-ascent perturbations with OBF perturbations | SAM on ViT-S/16 · ImageNet benchmarks and robustness datasets | top-1 accuracy: 78.73 -> 80.30; 85.47 -> 86.14; 66.98 -> 68.62; 25.69 -> 27.19; 24.10 -> 26.45 | Helped |
| Use OBF for finetuning instead of steepest-ascent SAM | ViT-S/32 pretrained without SAM · CIFAR-10 and CIFAR-100 | top-1 accuracy: 97.74 -> 97.79; 86.94 -> 87.04 | Helped |
| Pretrain with OBF instead of steepest-ascent SAM | ViT-S/32 transferred to CIFAR · CIFAR-10 and CIFAR-100 | top-1 accuracy: 97.69 -> 97.92; 86.21 -> 86.99 | Helped |
| Use dynamic α rather than fixed α (ablation) | OBF within SAM on ViT-S/32 · ImageNet-V1 | top-1 accuracy: 74.15 -> 74.53 | Helped |
| Use fixed α=0 instead of dynamic α (ablation) | OBF within SAM on ViT-S/32 · ImageNet-V1 | top-1 accuracy: 74.53 -> 73.33 | Hurt |
| Use fixed α=10^{-5} instead of dynamic α (ablation) | OBF within SAM on ViT-S/32 · ImageNet-V1 | top-1 accuracy: 74.53 -> 72.73 | Hurt |
| Use fixed α=10^{-4} instead of dynamic α (ablation) | OBF within SAM on ViT-S/32 · ImageNet-V1 | top-1 accuracy: 74.53 -> 72.49 | Hurt |
| Use fixed α=10^{-3} instead of dynamic α (ablation) | OBF within SAM on ViT-S/32 · ImageNet-V1 | top-1 accuracy: 74.53 -> 73.11 | Hurt |
| Use fixed α=10^{-2} instead of dynamic α (ablation) | OBF within SAM on ViT-S/32 · ImageNet-V1 | top-1 accuracy: 74.53 -> 73.65 | Hurt |
| Use fixed α=10^{-1} instead of dynamic α (ablation) | OBF within SAM on ViT-S/32 · ImageNet-V1 | top-1 accuracy: 74.53 -> 73.77 | Hurt |
| Use fixed α=1 instead of dynamic α (ablation) | OBF within SAM on ViT-S/32 · ImageNet-V1 | top-1 accuracy: 74.53 -> 74.15 | Hurt |
Surprising findings
- OBF-trained models often do not have the lowest sharpness; the flattest models seldom perform best on ImageNet.
- Forgetting-based correlation with generalization can exceed the correlation of flatness (sharpness) with accuracy for suitable binning thresholds.
Limitations
- Evaluation limited to image classification experiments
- OBF is not claimed to be optimal for all settings; optimal gamma and lambda differ by architecture
- GSAM results with ViT could not be fully reproduced against officially reported numbers
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100
Benchmarks: ImageNet-V1 ImageNet-Real ImageNet-V2 ImageNet-R ImageNet-Sketch CIFAR-10 CIFAR-100
Metrics: top-1 accuracy sharpness (dominant Hessian eigenvalue via power iteration) Kendall rank correlation
Baselines: SAM GSAM ASAM AdamW SGD ShrinkPerturb LabelSmoothing Random SAM
Builds on: sharpness-aware minimization GSAM ASAM m-SAM information bottleneck
Closest papers by meaning
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.90)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.89)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.88)
- Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise (NeurIPS 2025 · similarity 0.88)
- PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.88)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.