Research map / Architectures and efficiency / RNNs and normalization
Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply weight decay only during first ~100 epochs instead of throughout (ablation) | ResNet-18 / All-CNN training · CIFAR-10, CIFAR-100 | test accuracy: around 5% test error | No clear effect |
| remove weight decay shortly after critical period (e.g. epoch 75, All-CNN, no BN) (ablation) | All-CNN · CIFAR-10 | test accuracy | Helped |
| delay weight decay onset to epoch 50 (ablation) | ResNet-18 · CIFAR-10 | test error: 5% -> 7% | Hurt |
| apply regularization only late in training (after critical period) (ablation) | ResNet-18 / All-CNN · CIFAR-10, CIFAR-100 | test accuracy | No clear effect |
| 50-epoch weight decay window starting at epoch 0 vs epoch 50 (ablation) | ResNet-18 · CIFAR-10 | mean test accuracy increase: 0.24% -> 1.92% | Helped |
| remove batch normalization (ablation) | critical-period regularization experiments · CIFAR-10 | critical period effect | No clear effect |
| switch to piecewise constant learning rate schedule (ablation) | critical-period experiments · CIFAR-10 | critical period effect | No clear effect |
Closest papers by meaning
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.90)
- When to Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks (AAAI 2024 · similarity 0.87)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.87)
- Grokking at the Edge of Numerical Stability (ICLR 2025 · similarity 0.87)
- Weight decay induces low-rank attention layers (NeurIPS 2024 · similarity 0.86)
- When Will Gradient Regularization Be Harmful? (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.