Research map / Learning theory and algorithms / Prediction and decision losses
Tilted Sharpness-Aware Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tilted SAM (TSAM), a smoothed SAM variant using exponential tilting over perturbation neighborhoods, solved with an Euler-discretized HMC-inspired sampler, yielding flatter minima and better test accuracy than SAM and its variants on image and GLUE tasks.
Problem
Sharpness-aware minimization requires solving a min-max problem that is computationally hard for non-convex losses, and focusing only on the worst-case neighborhood point ignores many other high-loss local solutions, which can leave flat-minima search suboptimal.
Approach
Replace the max over perturbations in SAM with an exponentially tilted (log-mean-exp) average of losses over the perturbation ball, which reweights many neighborhood solutions by their loss, is smoother than SAM for bounded t, and recovers SAM as t grows; the gradient is estimated by sampling perturbations with a cheap one-step Euler/HMC-style sampler.
Key results
- TSAM reaches 77.78% on CIFAR100 with ResNet18 vs 76.52% for SAM and 71.39% for ERM, and 69.98% on noisy CIFAR100 vs 69.00% for SAM.
- On GLUE with DistilBERT, TSAM averages 78.01 vs 77.56 for SAM and 77.15 for ERM.
- Top-5 Hessian eigenvalues for TSAM (t=20) are lower (max 140.91) than SAM (max 232.60) and ERM (max 342.11) on CIFAR100 ResNet18.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TSAM instead of the strongest listed baseline | ResNet18 and WideResNet16-8 · CIFAR100 | accuracy: 76.52 -> 77.78; 80.22 -> 80.85 | Helped |
| Use TSAM instead of the strongest listed baseline | ViT · DTD | accuracy: 68.35 -> 68.82 | Helped |
| Use TSAM instead of the strongest listed baseline | WideResNet16-8 · DTD | accuracy: 18.23 -> 18.63 | Helped |
| Use TSAM instead of the strongest listed baseline | ResNet18 and WideResNet16-8 · Noisy CIFAR100 | accuracy: 69.31 -> 69.98; 69.79 -> 70.26 | Helped |
| Use TSAM instead of the strongest listed baseline | ResNet18 and ResNet34 · TinyImageNet | accuracy: 73.57 -> 73.55; 77.80 -> 77.79 | Mixed |
| Use TSAM instead of the strongest listed baseline | DistilBERT · GLUE | average score: 77.62 -> 78.01 | Helped |
| Increase sampled perturbations from s=3 to s=5 (ablation) | TSAM, ResNet18 · CIFAR100 | test accuracy: 77.40 -> 77.78 | Helped |
| Increase sampled perturbations from s=3 to s=5 (ablation) | TSAM, ViT · DTD | test accuracy: 68.82 -> 68.70 | Hurt |
| Increase HMC steps from N=1 to N=2 or N=3 (s=3) (ablation) | TSAM · CIFAR100 | test accuracy: 0.7740 -> 0.7745; 0.7740 -> 0.7752 | No clear effect |
| Increase HMC steps from N=1 to N=2 or N=3 (s=3) (ablation) | TSAM · DTD | test accuracy: 0.6882 -> 0.6883; 0.6882 -> 0.6880 | No clear effect |
| Schedule t instead of keeping t fixed (ablation) | TSAM, ResNet18 · Noisy CIFAR100 | test accuracy | No clear effect |
| Use TSAM instead of ERM or SAM | ResNet18 · CIFAR100, test loss | test loss: 0.9302 -> 0.7163; 0.91 -> 0.90 | Helped |
Surprising findings
- Naively sampling uniformly and performing tilted aggregation is worse than the proposed sampler even with 10 samples.
- Scheduling t (increasing or decreasing) does not significantly change results versus a fixed t.
Limitations
- Requires multiple gradient evaluations per iteration (s samples), increasing cost versus SAM; the sampler does not guarantee exact sampling from the tilted distribution.
- Optimal t must be tuned by grid search and is not known a priori.
- Theoretical flatness results are for generalized linear models and the connection between flatness and generalization remains open.
Experimental setup
Datasets: CIFAR100 DTD TinyImagenet GLUE ImageNet
Benchmarks: CIFAR100 DTD noisy CIFAR100 TinyImagenet GLUE
Metrics: test accuracy test loss top-5 Hessian eigenvalues neighborhood loss mean and variance GLUE standard metrics
Baselines: ERM SAM ESAM1 ESAM2 PGN RSAM
Builds on: sharpness-aware minimization hamiltonian monte carlo random smoothing
Closest papers by meaning
- Zeroth-Order Sharpness-Aware Learning with Exponential Tilting (NeurIPS 2026 · similarity 0.91)
- Improving Sharpness-Aware Minimization by Lookahead (ICML 2024 · similarity 0.90)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
- Lookbehind-SAM: k steps back, 1 step forward (ICML 2024 · similarity 0.89)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.89)
- Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.