Research map / Optimization / Implicit bias and generalization
CR-SAM: Curvature Regularized Sharpness-Aware Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CR-SAM, which adds a normalized Hessian trace curvature regularizer to Sharpness-Aware Minimization, computed efficiently via parallel finite differences, and shows gains on CIFAR and ImageNet across ResNet and ViT models.
Problem
One-step gradient ascent approximation in SAM becomes less accurate as training progresses because loss landscape non-linearity increases, and conventional Hessian curvature metrics (trace, top eigenvalue) are sensitive to gradient scale and inconsistent between train and test sets.
Approach
Introduce a normalized Hessian trace (trace divided by gradient norm) as a curvature measure consistent across train and test sets, and regularize SAM with a log-based curvature term that penalizes the Hessian trace and gradient norm at different strengths; the trace is estimated via finite differences of loss evaluations that can run in parallel.
Key results
- On CIFAR-100 with ResNet-18 and cutout, CR-SAM reaches 81.26% vs 80.15% for SAM and 79.23% for SGD.
- On ImageNet-1k, CR-SAM improves ResNet-50 to 77.64% vs 76.48% for SAM, and ViT-B/32 to 75.28% vs 73.51% for SAM.
- Model geometry on ResNet-18/CIFAR-100 shows CR-SAM lowers test-set normalized Hessian trace (884.95) relative to SAM (1193.87) and SGD (1674.89).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add curvature regularization to SAM | ResNet-18 · CIFAR-10 and CIFAR-100 | accuracy: 96.46 -> 96.95; 79.81 -> 80.76 | Helped |
| Add curvature regularization to SAM | ResNet-50 · ImageNet-1k | accuracy: 76.48 -> 77.64 | Helped |
| Add curvature regularization to SAM | ResNet-50 · ImageNet-C | accuracy: 46.03 -> 46.94 | Helped |
| Add curvature regularization to SAM | ResNet-50 · ImageNet-R | accuracy: 23.13 -> 23.48 | Helped |
| Add curvature regularization to SAM | ResNet-101 · ImageNet-1k | accuracy: 78.64 -> 79.12 | Helped |
| Add curvature regularization to SAM | ResNet-101 · ImageNet-C | accuracy: 51.27 -> 51.87 | Helped |
| Add curvature regularization to SAM | ResNet-101 · ImageNet-R | accuracy: 25.89 -> 26.37 | Helped |
| Add curvature regularization to SAM | ViT-S/32 · ImageNet-1k | accuracy: 70.23 -> 71.68 | Helped |
| Add curvature regularization to SAM | ViT-S/32 · ImageNet-C | accuracy: 45.78 -> 46.46 | Helped |
| Add curvature regularization to SAM | ViT-S/32 · ImageNet-R | accuracy: 21.12 -> 21.98 | Helped |
| Add curvature regularization to SAM | ViT-B/32 · ImageNet-1k | accuracy: 73.51 -> 75.28 | Helped |
| Add curvature regularization to SAM | ViT-B/32 · ImageNet-C | accuracy: 46.98 -> 48.12 | Helped |
Surprising findings
- Conventional Hessian trace and top Hessian eigenvalue decrease late in training even though the one-step approximation ratio indicates growing curvature, showing these metrics are gradient-scale dependent.
Limitations
- Gains are modest (often around 0.5 to 1 point) and rely on grid-searched rho, alpha and beta per configuration.
- Evaluated only on image classification with CNN and ViT backbones; no language or other domains tested.
- Theoretical PAC-Bayes result is given only as an informal sketch.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k ImageNet-C ImageNet-R
Metrics: top-1 accuracy gradient norm Hessian trace normalized Hessian trace
Baselines: SGD SAM R-SAM
Builds on: Sharpness-Aware Minimization Random SAM Hutchinson method PAC-Bayes bounds
Closest papers by meaning
- Avoiding spurious sharpness minimization broadens applicability of SAM (ICML 2025 · similarity 0.90)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.89)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.87)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.87)
- Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems (NeurIPS 2024 · similarity 0.87)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.