Research map / Optimization / Implicit bias and generalization
Why Do We Need Weight Decay in Modern Deep Learning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that weight decay plays different roles in two regimes: in multi-pass vision training (ResNet) it enables non-vanishing SGD noise that implicitly regularizes the Jacobian norm, while in one-pass LLM training (GPT-2 with AdamW) it acts as an effective learning-rate modulator that improves the bias-variance tradeoff and bfloat16 stability.
Problem
Weight decay is widely used in deep learning, but its role is poorly understood: classical regularization views do not explain its effect on over-parameterized ResNets trained for many epochs or on LLMs trained for about one epoch.
Approach
Weight decay combined with large learning rates stabilizes the training loss (loss stabilization), keeping SGD noise non-vanishing so implicit regularization of the Jacobian norm takes effect; in one-pass LLM training, weight decay instead raises the effective learning rate under sign-like Adam updates, improving bias-variance tradeoff and stability rather than acting as a regularizer.
Key results
- On CIFAR-5m ResNet18 with fixed 5M iterations, weight decay improves test error in both over-training and one-pass regimes.
- For GPT-2-small on OpenWebText, weight decay 0.1 and 0.3 lowers final training loss compared with lambda=0 (about 0.02 lower, as in prior work), and matching the effective learning rate without weight decay reproduces the training curves in float32.
- Without weight decay, bfloat16 training with learning rate 0.001 diverges late in training for all seeds, while weight decay yields stable bfloat16 training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WD 0.0015 rather than no WD with LR 0.15 | ResNet18 with EMA · CIFAR-10 test | test error: 0.13 -> 0.09 | Helped |
| Average SGD iterates as a snapshot ensemble instead of EMA | ResNet18 with WD 0.0075 and LR 0.05 · CIFAR-10 test | test error: 0.08 -> 0.09 | Mixed |
| Replace decoupled weight decay with an L2 penalty (ablation) | GPT-2-124M trained with AdamW · OpenWebText | training loss | No clear effect |
| Include LayerNorm parameters in weight decay (ablation) | GPT-2-124M · OpenWebText | training loss | Mixed |
| Use SGD with momentum instead of AdamW (ablation) | GPT-2-124M · OpenWebText | training loss | No clear effect |
| Remove weight decay (ablation) | GPT-2-124M trained with bfloat16 · OpenWebText, context length 1024 | training stability | Hurt |
| Add weight decay | GPT-2-124M trained with AdamW · OpenWebText | training loss: ≈0.02 lower | Helped |
| Match the weight-decay run's effective learning rate without weight decay (ablation) | GPT-2-124M trained in float32 · OpenWebText | training loss dynamics | No clear effect |
| Match effective learning rate without weight decay in bfloat16 (ablation) | GPT-2-124M · OpenWebText | training stability | Hurt |
| Reduce learning rate from 0.001 to 0.0006 instead of adding weight decay (ablation) | GPT-2-124M trained with bfloat16 · OpenWebText, context length 1024 | training speed: 0.001 to 0.0006 | Hurt |
| Use weight decay with LR 0.001 rather than no weight decay | GPT-2-124M trained with bfloat16 · OpenWebText, context length 1024 | training stability: 0.001 | Helped |
| Use EMA instead of learning-rate-decay fine-tuning (ablation) | ResNet18 · CIFAR-10 | test error | Helped |
Surprising findings
- Two runs with the same final parameter norm and same loss can generalize differently, so the norm itself is not what drives the benefit of weight decay.
- Matching the effective learning rate without weight decay reproduces the AdamW loss curve only in float32; bfloat16 runs still diverge, so weight decay remains necessary for stability.
Limitations
- Experiments are small scale due to limited compute; no truly large-scale runs.
- The noise-driven implicit regularization conjecture is not proven theoretically.
- The effective-learning-rate explanation for LLMs relies on a sign-SGD surrogate for Adam and is not derived for the full transformer.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet CIFAR-5m OpenWebText
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet OpenWebText validation loss
Metrics: test error training cross-entropy loss Jacobian norm parameter L2 norm total variation distance validation loss
Baselines: Adam without weight decay SGD without weight decay learning rate decay fine-tuning float32 training
Builds on: AdamW scale-invariant networks effective learning rate label-noise SGD implicit regularization NanoGPT
Closest papers by meaning
- Investigating the Role of Weight Decay in Enhancing Nonconvex SGD (CVPR 2025 · similarity 0.90)
- Weight decay induces low-rank attention layers (NeurIPS 2024 · similarity 0.90)
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.89)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.88)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.88)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.