Research map / Optimization / Optimizers
Decoupled Weight Decay Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Decouple weight decay from the gradient-based update | Adam · CIFAR-10 | test error: 15% relative improvement | Helped |
| Decouple weight decay from the gradient-based update | Adam · ImageNet32x32 | test error: 15% relative improvement | Helped |
| Decouple weight decay from the gradient-based update | Adam on EEG test data with Deep4Net · EEG test set | accuracy: 71.37% -> 73.68% | Helped |
| Decouple weight decay from the gradient-based update | Adam on EEG test data with ResNet · EEG test set | accuracy: 61.34% -> 72.04% | Helped |
| Use cosine annealing instead of a fixed or step-drop learning-rate schedule (ablation) | AdamW · CIFAR-10 | test error | Helped |
| Use decoupled weight decay instead of L2 regularization (ablation) | Adam · CIFAR-10, 100 epochs | Top-1 test error | Helped |
| Use a scheduled global learning-rate multiplier (ablation) | Adam · CIFAR-10 | test error | Helped |
| Decouple the weight-decay factor from the learning rate (ablation) | SGD with momentum · CIFAR-10 | Top-1 test error | Helped |
| Decouple the weight-decay factor from the learning rate (ablation) | Adam · CIFAR-10 | Top-1 test error | Helped |
| Add normalized weight decay based on batch size, dataset size, and training duration (ablation) | AdamW · CIFAR-10 and ImageNet32x32 | test error: roughly 5 times too large | Helped |
| Add cosine-annealing warm restarts (ablation) | AdamW · CIFAR-10 and ImageNet32x32 | speed: up to a factor of 10 | Helped |
| Use AdamW with warm restarts instead of AdamW without restarts (ablation) | AdamW · CIFAR-10 and ImageNet32x32 | anytime performance | Helped |
Closest papers by meaning
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.91)
- Investigating the Role of Weight Decay in Enhancing Nonconvex SGD (CVPR 2025 · similarity 0.90)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.89)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.89)
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.89)
- Weight decay induces low-rank attention layers (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.