Research map / Optimization / Optimizers
Sassha: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sassha, a second-order optimizer that combines sharpness-aware perturbation with a stabilized diagonal Hessian (square root, absolute value) and lazy Hessian updates, reporting better generalization than SGD, AdamW, SAM and other approximate second-order methods.
Problem
Approximate second-order optimizers often generalize worse than first-order methods, and the paper finds they converge to sharper minima than SGD.
Approach
Integrate a SAM-style sharpness-minimizing perturbation into a diagonal Hessian (Hutchinson) preconditioned optimizer, and stabilize the Hessian estimate with a square root and absolute value so it remains numerically stable and tolerates lazy Hessian reuse.
Key results
- On CIFAR-100 ResNet-32, Sassha reaches 72.14% validation accuracy vs 69.32% for SGD and 68.06% for AdaHessian.
- On Wikitext-2 GPT1-mini pretraining, Sassha achieves perplexity 122.40 vs 157.60 for Sophia-H and 175.06 for AdamW.
- Sassha uses about 2.3 gradient computations per step versus about 4 for conventional second-order methods, with lazy Hessian interval k=10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Sassha instead of SAM (SGD) | ResNet-20 · CIFAR-10 | accuracy: 92.85 ± 0.07 -> 92.98 ± 0.05 | Helped |
| use Sassha instead of SAM (SGD) | ResNet-32 · CIFAR-10 | accuracy: 93.89 ± 0.13 -> 94.09 ± 0.24 | Helped |
| use Sassha instead of SAM (SGD) | ResNet-32 · CIFAR-100 | accuracy: 71.99 ± 0.20 -> 72.14 ± 0.16 | Helped |
| use Sassha instead of SAM (SGD) | WRN-28-10 · CIFAR-100 | accuracy: 83.14 ± 0.13 -> 83.54 ± 0.08 | Helped |
| use Sassha instead of SAM (AdamW) | ViT-s-32 · ImageNet | accuracy: 68.31 ± 0.17 -> 69.20 ± 0.30 | Helped |
| use Sassha instead of Sophia-H | GPT1-mini · Wikitext-2 | perplexity: 157.60 ± 0.37 -> 122.40 ± 0.16 | Helped |
| add gradient momentum in the perturbation step (ablation) | Sassha · CIFAR-10 | accuracy: 92.98[0.05] -> 92.36[0.23] | Hurt |
| add gradient momentum in the perturbation step (ablation) | Sassha · CIFAR-100 | accuracy: 72.14[0.16] -> 70.93[0.21] | Hurt |
| add gradient momentum in the perturbation step (ablation) | Sassha · Wikitext-2 | perplexity: 122.40[0.16] -> 125.01[0.21] | Hurt |
| remove square-root scaling of the Hessian (ablation) | Sassha · ResNet-32/CIFAR-100 | training stability | Hurt |
| add the absolute-value function to Hessian estimates (ablation) | Sassha · training | training stability | Helped |
Surprising findings
- Sassha outperforms SAM on most workloads, with larger gains on Transformer models such as ViT and GPT/BERT.
- Naively removing the square root causes training to diverge in most runs, as sharpness minimization drives diagonal Hessian entries toward zero.
Limitations
- Theory covers only convergence to stationary points under convex, smooth assumptions; no convergence rate or generalization bound is provided.
- Experiments are limited to standard vision and small language tasks, not extreme-scale models or data.
- Compared SAM variants (ASAM, GSAM) require considerably more hyperparameter tuning.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet Wikitext-2 GLUE
Benchmarks: CIFAR-10 CIFAR-100 ImageNet Wikitext-2 GLUE
Metrics: validation accuracy perplexity sharpness (lambda_max, trace, delta L) wall-clock time
Baselines: SGD AdamW SAM AdaHessian Sophia-H Shampoo ASAM GSAM
Builds on: SAM AdaHessian Sophia AdamW Shampoo
Closest papers by meaning
- Lookbehind-SAM: k steps back, 1 step forward (ICML 2024 · similarity 0.88)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.88)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.87)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.87)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.87)
- Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.