Research map / Optimization / Implicit bias and generalization
Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the SDE framework to two-parameter (eta, rho) weak approximation to explain m-sharpness via stochastic gradient noise, and proposes Reweighted SAM (RW-SAM), which samples perturbation weights by per-sample gradient norm to match m-SAM while staying parallelizable.
Problem
Explaining why SAM generalizes better as the micro-batch size m used for perturbations decreases (m-sharpness), and obtaining that benefit without the sequential cost of m-SAM.
Approach
Derive SDE approximations for n-SAM, mini-batch SAM and m-SAM (USAM and normalized variants) showing that perturbation noise induces an implicit regularizer on the trace of the stochastic gradient noise covariance whose strength grows as m decreases; then reweight samples in the perturbation by a Gibbs distribution over per-sample gradient norms (RW-SAM), which is parallelizable.
Key results
- RW-SAM improves ResNet-18 on CIFAR-100 from 78.90 (SAM) to 79.31 and WideResNet on CIFAR-100 from 83.25 to 83.52.
- On ImageNet-1K with ResNet-50, RW-SAM reaches 77.37 vs 77.16 for SAM and 76.67 for SGD.
- On GLUE with DistilBERT, RW-SAM averages 0.809 vs 0.800 for SAM and 0.804 for AdamW; at 80% label noise on CIFAR-10 it gains about 16 points over SAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RW-SAM instead of SAM | ResNet-18, ResNet-50, and WideResNet-28-10 · CIFAR-10 test | accuracy: 95.99 ± 0.07 -> 96.24 ± 0.05; 96.06 ± 0.04 -> 96.34 ± 0.04; 96.91 ± 0.02 -> 97.11 ± 0.05 | Helped |
| Use RW-SAM instead of SAM | ResNet-18, ResNet-50, and WideResNet-28-10 · CIFAR-100 test | accuracy: 78.90 ± 0.27 -> 79.31 ± 0.28; 80.31 ± 0.35 -> 80.83 ± 0.05; 83.25 ± 0.07 -> 83.52 ± 0.08 | Helped |
| Use RW-SAM instead of SAM | ResNet-50 · ImageNet-1K test | accuracy: 77.16 ± 0.04 -> 77.37 ± 0.05 | Helped |
| Use RW-SAM instead of SAM | ViT-B/16 fine-tuning · CIFAR-10 | accuracy: 98.40 ± 0.02 -> 98.58 ± 0.02 | Helped |
| Use RW-SAM instead of SAM | ViT-B/16 fine-tuning · CIFAR-100 | accuracy: 89.63 ± 0.12 -> 89.89 ± 0.09 | Helped |
| Use RW-SAM instead of SAM | DistilBERT fine-tuning · GLUE average | average score: 0.800 -> 0.809 | Helped |
| Decrease the m-SAM micro-batch size | m-SAM · Generalization performance | generalization performance | Helped |
| Use deterministic full-batch perturbations rather than mini-batch perturbations | n-SAM · Generalization performance | generalization performance | Hurt |
| Use RW-SAM rather than SAM | ResNet-18 with randomly flipped CIFAR-10 labels · CIFAR-10 clean test, 80% label noise | accuracy: +16% absolute accuracy | Helped |
| Normalize estimated gradient norms before the exponential weighting (ablation) | RW-SAM · Hyperparameter sensitivity | performance sensitivity to λ | Helped |
| Use Rademacher rather than Gaussian perturbations for the Monte Carlo estimator (ablation) | RW-SAM · Gradient-norm estimation | estimator variance | Helped |
| Set the Monte Carlo sample count to Q=1 (ablation) | RW-SAM · Training experiments | performance improvement | Helped |
Surprising findings
- n-SAM (full-batch perturbation) gives little to no generalization gain over SGD, in contrast to mini-batch SAM.
- On CIFAR-100 with ResNet-18, RW-SAM and SAM are essentially tied on the baseline comparison (SAM 78.90 vs SGD 78.91).
Limitations
- Theory relies on assumptions such as log-concave per-sample gradients, near-orthogonality of full gradient and SGN, and Lipschitz conditions; normalized SAM lacks closed-form drift.
- RW-SAM needs a hyperparameter lambda and finite-difference step delta tuned per model.
- Gradient-norm estimation via finite differences adds an extra forward pass per step.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K GLUE
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K GLUE
Metrics: test accuracy GLUE score
Baselines: SGD SAM AdamW m-SAM
Builds on: Sharpness-Aware Minimization Unnormalized SAM (USAM) SDE framework for SAM (Compagnoni et al. style weak approximation) m-sharpness (Foret et al.)
Closest papers by meaning
- Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization (NeurIPS 2024 · similarity 0.92)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.91)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.91)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.90)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.90)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.