Research map / Optimization / Implicit bias and generalization
A Universal Class of Sharpness-Aware Minimization Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a universal parameterized class of sharpness measures S(x;phi,psi,mu) over the training-loss Hessian, with matching sharpness-aware objectives, scale-invariant instances, and Frob-SAM and Det-SAM algorithms evaluated on CIFAR10/100 and SVHN.
Problem
Existing sharpness measures (max eigenvalue, trace of the training-loss Hessian) are not meaningful for non-convex neural networks and are not invariant to parameter rescalings, and only a few explicit sharpness-aware biases are known.
Approach
Represent any sharpness measure as phi of an integral of psi applied to the quadratic form v^T H v under a measure mu; prove universality over Hessian eigenvalues, derive a zeroth-order sharpness-aware loss with explicit bias toward that measure, and choose mu (e.g., Lebesgue measure) to get scale invariance, yielding Frob-SAM and Det-SAM.
Key results
- On CIFAR10 with 20% corrupted labels, Frob-SAM reaches 87.57% test accuracy versus 85.42% for SAM and 82.78% for SGD.
- On the 10%-subsampled CIFAR10-S, Frob-SAM reaches 74.49% versus 72.96% for SAM.
- On MNIST, the Frobenius norm of the Hessian decreases throughout Frob-SAM training, with larger regularization lambda giving lower norm; all settings exceed 94% test accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Frob-SAM instead of SAM | ResNet18 · CIFAR10-S | test accuracy: 72.96 -> 74.49 | Helped |
| Use Frob-SAM instead of SAM | ResNet18 · CIFAR100-S | test accuracy: 36.87 -> 38.30 | Helped |
| Use Frob-SAM instead of SAM | ResNet18 · SVHN-S | test accuracy: 89.80 -> 89.23 | Hurt |
| Use Frob-SAM instead of SAM | ResNet18 · CIFAR10-C | test accuracy: 85.42 -> 87.57 | Helped |
| Use Frob-SAM instead of SAM | ResNet18 · CIFAR100-C | test accuracy: 66.12 -> 65.73 | Hurt |
| Use Frob-SAM instead of SAM | ResNet18 · SVHN-C | test accuracy: 90.66 -> 90.11 | No clear effect |
| Increase the Frob-SAM regularization strength λ | Frob-SAM · MNIST | Hessian Frobenius norm | Helped |
Surprising findings
- Det-SAM is weaker than other variants on the corrupted-label CIFAR10-C (83.61% versus 87.57% for Frob-SAM), and no single sharpness measure dominates across datasets.
- On the full clean datasets, SAM and ASAM often match or beat the proposed Frob/Trace/Det variants (e.g., CIFAR10: SSAM 95.78%, Frob 94.96%).
Limitations
- Universality with m=d requires large dimension of the hyperparameter; arbitrary functions of the Hessian need m up to d(d+1)/2, which is impractical for overparameterized models.
- Det-SAM requires truncating the Lebesgue integral to a hypercube, an approximation; the hyperparameter t is set with little tuning.
- No single sharpness measure is best across tasks, and the best bias a priori is unclear.
Experimental setup
Datasets: CIFAR10 CIFAR100 SVHN MNIST
Benchmarks: CIFAR10 CIFAR100 SVHN CIFAR10-S CIFAR100-S SVHN-S CIFAR10-C CIFAR100-C SVHN-C
Metrics: test accuracy Frobenius norm of Hessian (MNIST training)
Baselines: SGD SAM ASAM Sparse SAM (SSAM) Trace-SAM
Builds on: Sharpness-Aware Minimization Adaptive SAM average-based sharpness-aware objective
Closest papers by meaning
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.93)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.91)
- Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization (NeurIPS 2024 · similarity 0.91)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.91)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.90)
- Lookbehind-SAM: k steps back, 1 step forward (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.