Research map / Optimization / Implicit bias and generalization
Understanding the Disharmony between Weight Normalization Family and Weight Decay
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed weight decay with AWS on WS convolutions | WS-ResNet50 SGD · ImageNet val | top-1 accuracy: 76.74 -> 76.59 | No clear effect |
| replace fixed weight decay with AWS on WS convolutions | WS-ResNeXt-101 SGD · ImageNet val | top-1 accuracy: 78.68 -> 78.74 | No clear effect |
| use AWS instead of fixed weight decay under Adam (WS) | WS-ResNet50 · ImageNet val | training stability / top-1: failed (–) -> 72.55 | Helped |
| use AWS instead of fixed weight decay with large decay values | WS-ResNet50 SGD · ImageNet val | training stability: – -> 72.15 (alpha 1e-1) | Helped |
| remove weight decay (lambda=0) on WS convolutions (ablation) | WS-ResNet50 · ImageNet train | top-1 accuracy | Hurt |
| scale initial weights by 20 or 100 with fixed WD (ablation) | WS-ResNet50 · ImageNet | top-1 accuracy | Hurt |
| use AWS instead of fixed WD under scaled initialization (ablation) | WS-ResNet50 · ImageNet | top-1 accuracy | Helped |
Closest papers by meaning
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.89)
- Investigating the Role of Weight Decay in Enhancing Nonconvex SGD (CVPR 2025 · similarity 0.88)
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.87)
- Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries (ICLR 2025 · similarity 0.86)
- Weight decay induces low-rank attention layers (NeurIPS 2024 · similarity 0.86)
- When, Where and Why to Average Weights? (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.