Research map / Optimization / Optimizers
When Will Gradient Regularization Be Harmful?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows gradient regularization (GR) harms adaptive optimizers (Adam, RMSProp) when combined with learning rate warmup on ViTs, and proposes three GR warmup strategies (r-warmup, λ-warmup, zero-warmup) that fix it.
Problem
Gradient regularization penalizes gradient norm, but when combined with learning rate warmup under adaptive optimizers (Adam, RMSProp) it can degrade performance, especially for larger models trained from scratch.
Approach
Attribute the degradation to GR causing a decreasing gradient norm early in training, which inflates the variance of the adaptive learning rate; then ramp or delay GR during warmup (r-warmup, λ-warmup, or zero-warmup where GR is off during warmup) so gradient second-moment statistics accumulate accurately.</303>
Key results
- On ViT-B with Adam and LR warmup, zero-warmup GR reduces CIFAR-10 test error from 12.36% (vanilla GR) to 9.42%, roughly 3 points.
- Vanilla GR with LR warmup can be worse than LR warmup alone; e.g., ViT-B Adam on CIFAR-10 gives 12.36% with GR versus 10.83% without GR under warmup.
- All three GR warmup strategies improve over baseline GR across ViT, Swin and CaiT models on CIFAR and TinyImageNet, and zero-warmup gives the best results in most cases (e.g., ImageNet Swin-Tiny 22.34% to 21.58%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add zero-warmup GR instead of standard GR (ablation) | ViT-B with Adam · Cifar-10 | test error rate: 12.36_{ \pm 0.33} -> 9.42_{ \pm 0.15} | Helped |
| Add r-warmup GR instead of standard GR (ablation) | ViT-B with Adam · Cifar-10 | test error rate: 12.36_{ \pm 0.33} -> 10.22_{ \pm 0.26} | Helped |
| Add λ-warmup GR instead of standard GR (ablation) | ViT-B with Adam · Cifar-10 | test error rate: 12.36_{ \pm 0.33} -> 9.98_{ \pm 0.23} | Helped |
| Add zero-warmup GR instead of standard GR (ablation) | ViT-B with Adam · Cifar-100 | test error rate: 36.65_{ \pm 0.26} -> 35.75_{ \pm 0.20} | Helped |
| Add GR while using LR warmup | ViT-B with Adam · Cifar-10 | test error rate: 10.83_{ \pm 0.28} -> 12.36_{ \pm 0.33} | Hurt |
| Add standard GR without LR warmup | ViT models with Adam and RMSProp · Cifar-10 and Cifar-100 | test error rate | Helped |
| Add LR warmup | ViT models with Adam and RMSProp · Cifar-10 and Cifar-100 | test error rate | Helped |
| Use r-warmup rather than standard GR (ablation) | ViT models with Adam and RMSProp · Cifar-10 and Cifar-100 | test error rate | Helped |
| Use λ-warmup rather than standard GR (ablation) | ViT models with Adam and RMSProp · Cifar-10 and Cifar-100 | test error rate | Helped |
| Use zero-warmup rather than standard GR (ablation) | ViT, Swin, and CaiT models · Cifar-10 and Cifar-100 | test error rate | Helped |
| Use zero-warmup instead of vanilla GR (ablation) | ViT models with Adam · TinyImageNet | test error rate | Helped |
| Use zero-warmup instead of vanilla GR (ablation) | Swin-Tiny with Adam · ImageNet | test error rate: 22.34_{ \pm 0.15} -> 21.58_{ \pm 0.34} | Helped |
Surprising findings
- Adding GR can hurt performance when combined with LR warmup, even though GR helps when no warmup is used.
- Larger models (ViT-B) are more affected by the GR and warmup incompatibility.
Limitations
- Theoretical analysis relies on a simplified Gaussian gradient model with exponentially decaying variance and a Taylor approximation of the adaptive learning rate variance.
- Gains of r-warmup and lambda-warmup are relatively modest; the paper's experiments are limited to image classification with ViT-family models trained from scratch.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet ImageNet
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet ImageNet
Metrics: test error rate [%]
Baselines: Vanilla GR No GR Adam RMSProp
Builds on: Sharpness-Aware Minimization Adam RMSProp Vision Transformer learning rate warmup
Closest papers by meaning
- Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias (ICML 2023 · similarity 0.88)
- Loss Landscapes are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent (ICLR 2023 · similarity 0.87)
- Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training (NeurIPS 2024 · similarity 0.87)
- Why Warmup the Learning Rate? Underlying Mechanisms and Improvements (NeurIPS 2024 · similarity 0.85)
- How Memory in Optimization Algorithms Implicitly Modifies the Loss (NeurIPS 2025 · similarity 0.85)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.