Research map / Optimization / Implicit bias and generalization
A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops a linear stability framework for SGD, random-perturbation SGD, and SAM around minima, linking data coherence (alignment of per-example Hessians) to stability and an emergent simplicity bias, validated on quadratic toy problems and two-layer ReLU networks.
Problem
Why SGD and its variants (random perturbation, SAM) prefer flat and simple minima and how data structure governs which minima are stable attractors under training dynamics.
Approach
Defines a data-dependent coherence matrix from pairwise traces of per-example Hessians and shows its spectrum governs linear stability for SGD and SAM; SAM's effective Hessian (I + (rho/alpha)H)H penalizes high-curvature directions more strictly, amplifying preference for coherent, lower-complexity solutions.
Key results
- On synthetic two-layer ReLU training, SAM with rho=0.1 reduces lambda_max(S) from 12740 (SGD) to 3446 and effective rank from 6.167 to 5.6.
- On CIFAR-10 ResNet-18 (approximate coherence on 100 samples), feature rank falls from 148.33 (SGD) to 128.67 (SAM rho=0.2) while approximate coherence rises from 1.0045 to 1.0907.
- Theory predicts SAM divergence threshold is stricter than SGD and lower-complexity (smaller C) generalizing solutions have smaller coherence spectrum, hence are more stable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase SAM radius from ρ = 0.01 to ρ = 0.1 (ablation) | SAM on the two-layer ReLU network · Two-layer ReLU network training | Coherence Measure: 121.473 -> 65.656 | Hurt |
| Increase SAM radius from ρ = 0.01 to ρ = 0.1 (ablation) | SAM on the two-layer ReLU network · Two-layer ReLU network training | ER: 6.2 -> 5.6 | Mixed |
| Use SAM with ρ = 0.1 instead of SGD (ablation) | Two-layer ReLU network · Two-layer ReLU network training | Coherence Measure: 133.942 -> 65.656 | Hurt |
| Use SAM with ρ = 0.1 instead of SGD (ablation) | Two-layer ReLU network · Two-layer ReLU network training | ER: 6.167 -> 5.6 | Mixed |
| Use SAM with radius 0.05 instead of SGD (ablation) | ResNet-18 · CIFAR-10 | Rank: 148.33 ± 3.22 -> 157.67 ± 9.29 | Hurt |
| Use SAM with radius 0.2 instead of SGD (ablation) | ResNet-18 · CIFAR-10 | Rank: 148.33 ± 3.22 -> 128.67 ± 5.51 | Helped |
| Use SAM with radius 0.2 instead of SGD (ablation) | ResNet-18 · CIFAR-10 | Coherence: 1.0045 ± 0.0020 -> 1.0907 ± 0.0890 | Mixed |
| Initialize near a lower-C rather than higher-C (C,r)-generalizing solution (ablation) | SGD and SAM · Two-layer ReLU network local stability | Convergence speed | Helped |
| Use SAM rather than SGD | Two-layer ReLU network near (C,r)-generalizing solutions · Local stability | Convergence speed | Helped |
| Add random perturbation to SGD | SGD · Quadratic-loss local stability | Stability threshold | No clear effect |
| Add random perturbation to SGD | SGD · Quadratic-loss local stability | Divergence speed | Helped |
| Use SAM rather than SGD | Optimization near minima · Local stability | Stability of sharp minima | Hurt |
Surprising findings
- The paper reports that coherence rises for SAM on CIFAR-10 approximation even though the synthetic experiments show coherence falling, which conflicts with the synthetic-setting narrative.
Limitations
- Analysis is local and relies on a linear/quadratic approximation near minima.
- Uses i.i.d. data and restricted (C,r)-generalizing solutions for two-layer ReLU networks.
- Coherence measure is computationally expensive and not a practical training diagnostic; large-scale results are only approximate.
Experimental setup
Datasets: synthetic binary inputs with y = x[0]x[1] CIFAR-10
Benchmarks: CIFAR-10
Metrics: coherence measure lambda_max(S) effective rank of features (90% PCA variance) weight norm divergence/convergence convergence speed
Baselines: SGD random perturbation SGD
Builds on: sharpness-aware minimization linear stability analysis of SGD simplicity bias
Closest papers by meaning
- The alignment property of SGD noise and how it helps select flat minima: A stability analysis (NeurIPS 2022 · similarity 0.92)
- A Precise Characterization of SGD Stability Using Loss Surface Geometry (ICLR 2024 · similarity 0.92)
- Stability Analysis of Sharpness-Aware Minimization (ICML 2026 · similarity 0.91)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.91)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.90)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.