Research map / Learning theory and algorithms / Prediction and decision losses
Improving SAM Requires Rethinking its Optimization Formulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates Sharpness-Aware Minimization as a bilevel problem in which the perturbation player maximizes a differentiable lower bound on the 0-1 loss (BiSAM), gaining small accuracy gains over SAM at similar cost.
Problem
SAM's inner maximization uses an upper-bound surrogate (cross-entropy), so the adversarial weight perturbation may not increase misclassification error, yielding weaker perturbations.
Approach
Replace the upper-bound surrogate in the perturbation (ascent) step with a differentiable log-sum-exp lower bound on the 0-1 loss built from a lower-bound step approximation (tanh or a shifted softplus), keeping the minimizer on cross-entropy; this gives a bilevel formulation solved with a SAM-style first-order step at the same cost.
Key results
- On CIFAR-10, BiSAM (-log) averages 95.88% vs SAM 95.78% across five models; on CIFAR-100 average 78.78 (tanh) / 78.78 (-log) vs SAM 78.52.
- On ImageNet-1K, BiSAM (-log) reaches 75.96% top-1 vs SAM 75.83%.
- BiSAM (-log) improves BERT-base GLUE average to 77.04 vs SAM 75.60 (AdamW 73.21).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SAM's cross-entropy perturbation objective with the BiSAM -log lower-bound objective | SAM · CIFAR-10 test, 5-model average | accuracy: 95.78_{\pm 0.06} -> 95.88_{\pm 0.08} | Helped |
| Replace SAM's cross-entropy perturbation objective with the BiSAM -log lower-bound objective | SAM · CIFAR-100 test, 5-model average | accuracy: 78.52_{\pm 0.13} -> 78.78_{\pm 0.09} | Helped |
| Replace SAM's cross-entropy perturbation objective with the BiSAM -log lower-bound objective | SAM · ImageNet-1K test | Top1 accuracy: 75.83_{\pm 0.16} -> 75.96_{\pm 0.15} | Helped |
| Use tanh instead of the -log lower bound (ablation) | BiSAM · CIFAR-10 test, 5-model average | accuracy: 95.88_{\pm 0.08} -> 95.93_{\pm 0.08} | Helped |
| Use tanh instead of the -log lower bound (ablation) | BiSAM · ImageNet-1K test | Top5 accuracy: 92.49_{\pm 0.10} -> 92.40_{\pm 0.13} | Hurt |
| Add adaptive sharpness to the BiSAM perturbation objective | ASAM · CIFAR-10 test, 5-model average | accuracy: 96.07_{\pm 0.05} -> 96.14_{\pm 0.05} | Helped |
| Replace the ASAM perturbation objective with the BiSAM -log lower-bound objective | ASAM · CIFAR-10 test, DenseNet-121 | accuracy: 96.79_{\pm 0.14} -> 96.79_{\pm 0.13} | No clear effect |
| Add ESAM's SWP and SDS to the BiSAM method | ESAM · CIFAR-10 test, 5-model average | accuracy: 95.56_{\pm 0.09} -> 95.71_{\pm 0.06} | Helped |
| Replace the ESAM perturbation objective with the BiSAM -log lower-bound objective | ESAM · CIFAR-10 test, WRN-28-10 | accuracy: 97.17_{\pm 0.09} -> 97.18_{\pm 0.05} | Helped |
| Replace SAM's perturbation objective with the BiSAM -log lower-bound objective | SAM with ViT-B/16 fine-tuning · Oxford-IITPets test | accuracy: 93.66_{\pm 0.48} -> 94.15_{\pm 0.24} | Helped |
| Add BiSAM to AdamW fine-tuning | AdamW with BERT-base · GLUE | aggregate score: 73.21 -> 77.04 | Helped |
| Replace SAM's perturbation objective with the BiSAM -log lower-bound objective | SAM with ResNet-32 · CIFAR-10 test, 40% label noise | accuracy: 89.03_{\pm 0.09} -> 88.71_{\pm 0.23} | Hurt |
Surprising findings
- Improvements over SAM on CIFAR and ImageNet are small (often within one standard deviation), while the paper emphasizes consistency.
- Reported per-epoch timing overhead for BiSAM (logsumexp) over SAM ranges up to roughly 10-30% in PyTorch, contrary to the claim of identical complexity.
Limitations
- The bilevel problem is solved by a first-order Taylor approximation of the inner problem, so the inner solution is not exact; the link between the BiSAM solution and the 0-1 minimax problem is left open.
- Gains over SAM are modest and sometimes within noise; hyperparameters mu and alpha tuned on CIFAR-10 with ResNet-56.
- Wall-clock overhead is not fully eliminated in PyTorch implementation.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K Oxford Flowers Oxford-IIIT Pets GLUE CIFAR-10 with label noise
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K Oxford Flowers Oxford-IIIT Pets GLUE CIFAR-10 label noise
Metrics: test accuracy top-1 accuracy top-5 accuracy Matthews correlation F1
Baselines: SGD SAM ASAM ESAM GSAM
Builds on: Sharpness-Aware Minimization ASAM ESAM mSAM
Closest papers by meaning
- Lookbehind-SAM: k steps back, 1 step forward (ICML 2024 · similarity 0.87)
- Zeroth-Order Sharpness-Aware Learning with Exponential Tilting (NeurIPS 2026 · similarity 0.87)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.87)
- Tilted Sharpness-Aware Minimization (ICML 2025 · similarity 0.87)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.86)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.