Research map / Label-efficient and robust learning / Noisy labels
Why is SAM Robust to Label Noise?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why 1-SAM is robust to random label noise, decomposing its effect into a logit-scale reweighting (explicit up-weighting of clean examples) and a network-Jacobian effect, and shows a cheap Jacobian-inspired weight/feature norm regularizer recovers part of the gains.
Problem
Sharpness-Aware Minimization (SAM) yields large test accuracy gains under random label noise, but the mechanism is unclear, especially since peak accuracy occurs early in training, not at convergence.
Approach
Decomposes per-sample SAM gradients into logit-scale and network-Jacobian terms; shows in linear models SAM up-weights low-loss (clean) examples via the logit scale, but in deep networks the Jacobian component (J-SAM) accounts for most robustness. For 2-layer linear networks, J-SAM reduces to SGD with L2 penalties on activations and last-layer weights, and a simple regularized SGD recovers part of
Key results
- On CIFAR10 with 30% label noise, 1-SAM (rho=0.01) reaches 69.47% best test accuracy vs 52.48% for SGD, while J-SAM reaches 69.17% and L-SAM only 54.13%.
- Adding the proposed last-layer weight and activation norm regularization to SGD raises best test accuracy to 60.8% (about 8 points over SGD), closing the gap to SAM from 17% to 9%.
- In linear toy Gaussian data with 40% label noise, early-stopping test accuracy of SAM increases monotonically with rho in the grid searched from 0 to 0.18.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 1-SAM instead of SGD | ResNet18 · CIFAR10, 30% label noise | best test accuracy: 52.48% -> 69.47% | Helped |
| Apply the SAM perturbation only to the Jacobian term (ablation) | 1-SAM · CIFAR10, 30% label noise | best test accuracy: 52.48% -> 69.17% | Helped |
| Apply the SAM perturbation only to the logit scale term (ablation) | 1-SAM · CIFAR10, 30% label noise | best test accuracy: 52.48% -> 54.13% | Helped |
| Add regularization on last-layer weights and logits | SGD · CIFAR10, 30% label noise | best test accuracy: 52.48% -> 60.8% | Helped |
| Increase the perturbation magnitude ρ (ablation) | 1-SAM · Toy Gaussian data, 40% label noise | early-stopping test accuracy | Helped |
| Use naive SAM instead of per-example 1-SAM (ablation) | Linear model · Toy Gaussian data, 40% label noise | early-stopping test accuracy | Hurt |
| Use SGD with last-layer weight and feature-norm regularization instead of SAM (ablation) | ResNet18 · CIFAR10, no label noise | test accuracy: 1% | No clear effect |
| Use regularized SGD instead of SAM (ablation) | ResNet18 · CIFAR10, 30% label noise | test accuracy: 17% to 9% | Hurt |
| Increase the number of training examples (ablation) | SAM relative to SGD · CIFAR10, 30% label noise | test accuracy gap | Helped |
| Increase model width (ablation) | ResNet18 · CIFAR10, 30% label noise | test accuracy | Helped |
| Increase the learning rate (ablation) | ResNet18 with SGD or SAM · CIFAR10, 30% label noise | test accuracy | Helped |
| Increase the learning rate (ablation) | Linear model with SGD or SAM · Toy Gaussian data, 40% label noise | early-stopping accuracy | Helped |
Surprising findings
- Removing SAM's explicit logit-scale reweighting (L-SAM vs J-SAM) in deep networks leaves label noise robustness nearly unchanged, contrary to the intuition from linear models.
- Naive SAM (n-SAM) gives little early-stopping gain over SGD unless rho is scaled up dramatically (e.g., 100000), while 1-SAM gives gains at standard rho.
Limitations
- The proposed regularization is explicitly described as illustrative and not a standalone method for label noise robustness; it does not fully match SAM.
- Theoretical Jacobian analysis is limited to 2-layer linear networks; deeper networks are studied empirically only.
- Experiments focus on synthetic label noise on CIFAR10 with ResNet18 using layer norm in place of batch norm.
Experimental setup
Datasets: CIFAR10 Tiny-ImageNet Flowers102 toy Gaussian data
Benchmarks: CIFAR10 Tiny-ImageNet Flowers102 toy Gaussian data
Metrics: best test accuracy early-stopping test accuracy clean/noisy training accuracy gap
Baselines: SGD L-SAM J-SAM 1-SAM n-SAM
Builds on: Sharpness-Aware Minimization 1-SAM (per-example perturbation) Arpit et al. memorization / clean-first learning weight decay implicit regularization
Closest papers by meaning
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.90)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.90)
- Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise (NeurIPS 2025 · similarity 0.89)
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.89)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
- Avoiding spurious sharpness minimization broadens applicability of SAM (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.