Research map / Probabilistic and scientific modeling / Molecules and proteins

Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization

NeurIPS 2024 · Probabilistic and scientific modeling › Molecules and proteins · approach: Sparse graphical and tensor models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes SAM dynamics with a third-order SDE showing top-Hessian-eigenvalue regularization depends on perturbation-eigenvector alignment, and proposes Eigen-SAM, which adds the gradient-orthogonal component of an estimated top eigenvector to the SAM perturbation.

Problem

SAM's implicit bias and its theory are poorly understood; existing second-order SDE analyses do not explain how SAM regularizes sharpness measured by the top Hessian eigenvalue, and alignment between SAM's perturbation and the top eigenvector is often poor in practice.

Approach

Derives a third-order SDE for SAM whose cubic drift term equals the gradient of the top Hessian eigenvalue under perturbation-eigenvector alignment, and proposes Eigen-SAM, which periodically estimates the top eigenvector via power iteration and adds its gradient-orthogonal component to the perturbation to explicitly enforce alignment.

Key results

• Eigen-SAM with ResNet-18 reaches 95.9% on CIFAR-10 and 78.3% on CIFAR-100, versus 95.5% and 77.4% for SAM.
• On WideResNet-28-10, Eigen-SAM achieves 96.8% on CIFAR-10 and 82.8% on CIFAR-100, versus 96.5% and 82.0% for SAM.
• Fine-tuning ViT-B-16 on CIFAR-100, Eigen-SAM reaches 89.8% versus 89.5% for SAM and 88.6% for SGD.

Measured effects reported in the paper

Surprising findings

• Perturbation-eigenvector alignment is often poor in practice, so SAM fails to efficiently minimize the top Hessian eigenvalue.
• Explicit gradient regularization with coefficient rho and Reverse-SAM each perform noticeably worse than SAM, contradicting the second-order SDE's prediction that EGR should match SAM.

Limitations

• Theory relies on strong regularity assumptions and an eigenvalue-gap assumption for differentiability of the top eigenvalue.
• Eigen-SAM adds Hessian-vector product overhead (roughly 2 + q/p to 2 + 2q/p times SGD cost).
• Experiments are limited to image classification and a single ViT fine-tuning setting.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 Fashion-MNIST SVHN MNIST

Benchmarks: CIFAR-10 CIFAR-100 Fashion-MNIST SVHN

Metrics: test accuracy Hessian top eigenvalue perturbation-eigenvector alignment approximation error vs discrete SAM

Baselines: SGD SAM Explicit Gradient Regularization Reverse-SAM second-order SDE for SAM

Builds on: Sharpness-Aware Minimization second-order SDE for SAM (Compagnoni et al.) SDE approximation of SGD (Li et al.) general PAC-Bayes theorem

Closest papers by meaning

• Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise (NeurIPS 2025 · similarity 0.92)
• A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.91)
• Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
• Zeroth-Order Sharpness-Aware Learning with Exponential Tilting (NeurIPS 2026 · similarity 0.89)
• Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems (NeurIPS 2024 · similarity 0.89)
• Tilted Sharpness-Aware Minimization (ICML 2025 · similarity 0.89)