Research map / Architectures and efficiency / RNNs and normalization
Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that SAM implicitly drives balancedness (squared-norm difference between two factor groups) toward zero in scale-invariant NOP/OP problems, more strongly with noisy data, and derives BAR, a cheap norm-based regularizer that replaces SAM's second gradient for LoRA finetuning.
Problem
Understanding global implicit regularization of sharpness-aware minimization (SAM) for scale-invariant two-group problems (e.g., LoRA, softmax attention), and why SAM helps under data anomalies, while making SAM's benefit available at low cost.
Approach
Replaces sharpness with balancedness (half the difference of squared norms of the two variable groups) as the implicit-regularization metric; shows SAM's balancedness dynamics are driven by gradient-norm differences and are data-responsive; converts this into an explicit, second-gradient-free regularizer (BAR) applied per LoRA layer.
Key results
- LoRA-oBAR and LoRA-nBAR improve few-shot RoBERTa-large average accuracy over LoRA (80.0) to 80.9 and 81.2, with LoRA-SAM at 81.3.
- On OPT-1.3B few-shot, LoRA-nBAR averages 79.2 vs LoRA 77.6 and LoRA-SAM 78.4, while BAR runtime is about 1.03-1.05x LoRA versus 3.28-4.43x for LoRA-SAM.
- On WebNLG with GPT2-medium, LoRA-nBAR reaches 55.20 BLEU vs LoRA 54.99.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add nBAR balancedness regularization to LoRA | LoRA on RoBERTa-large · RoBERTa-large few-shot average | accuracy: 80.0 -> 81.2 | Helped |
| Add oBAR balancedness regularization to LoRA | LoRA on RoBERTa-large · RoBERTa-large few-shot average | accuracy: 80.0 -> 80.9 | Helped |
| Add nBAR balancedness regularization to LoRA | LoRA on OPT-1.3B · OPT-1.3B few-shot average | accuracy: 77.6 -> 79.2 | Helped |
| Add oBAR balancedness regularization to LoRA | LoRA on OPT-1.3B · OPT-1.3B few-shot average | accuracy: 77.6 -> 78.5 | Helped |
| Add nBAR balancedness regularization to LoRA | LoRA on RoBERTa-large · RoBERTa-large finetuning average | accuracy: 85.3 -> 85.9 | Helped |
| Add oBAR balancedness regularization to LoRA | LoRA on RoBERTa-large · RoBERTa-large finetuning average | accuracy: 85.3 -> 85.7 | Helped |
| Add nBAR balancedness regularization to LoRA | LoRA on GPT2-medium · WebNLG | BLEU: 54.99±{0.24} -> 55.20±{0.16} | Helped |
| Add oBAR balancedness regularization to LoRA | LoRA on GPT2-medium · WebNLG | BLEU: 54.99±{0.24} -> 55.15±{0.19} | Helped |
| Use BAR instead of SAM | LoRA on OPT-1.3B · OPT-1.3B runtime | runtime: 4.43x -> 1.05x | Helped |
| Use BAR instead of SAM | LoRA on OPT-1.3B · OPT-1.3B runtime on CB | runtime: 3.34x -> 1.03x | Helped |
Surprising findings
- Directly imposing the hard constraint ||x||=||y|| can trap SGD at saddle points, so regularization is preferred over a constraint.
Limitations
- BAR is designed for scale-invariant modules and does not apply to linear models such as logistic regression.
- Gains on larger models were not verified due to hardware constraints.
- Theory relies on infinitesimal step size and well-defined SAM iterates; empirical validation uses eta=0.1.
Experimental setup
Datasets: SST-2 SST-5 SNLI MNLI RTE TREC CB COPA ReCoRD SQuAD STS-B WebNLG
Benchmarks: SST-2 SST-5 SNLI MNLI RTE TREC CB COPA ReCoRD SQuAD STS-B WebNLG
Metrics: accuracy F1 BLEU runtime overhead
Baselines: SGD LoRA LoRA-SAM prefix tuning full finetuning adapter
Builds on: SAM LoRA AdamW prefix tuning
Closest papers by meaning
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning (ICLR 2024 · similarity 0.93)
- Sharpness-Aware Minimization: General Analysis and Improved Rates (ICLR 2025 · similarity 0.91)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.90)
- Friendly Sharpness-Aware Minimization (CVPR 2024 · similarity 0.90)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.89)
- Avoiding spurious sharpness minimization broadens applicability of SAM (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.