Research map / Optimization / Implicit bias and generalization
On the training dynamics of deep networks with $L_2$ regularization
Aitor Lewkowycz, Guy Gur-Ari
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train for a number of steps proportional to 1/lambda instead of a fixed budget while decreasing L2 | overparameterized networks (WRN, FC, CNN) · CIFAR-10 / MNIST | test accuracy | Improved |
| reduce L2 parameter beyond a certain value under a fixed training budget | overparameterized networks · CIFAR-10 / MNIST | test accuracy | Hurt |
| use predicted L2 (from measured c) instead of tuned L2 | WRN28-10 with momentum, augmentation · CIFAR-10 | test accuracy: 0.967 -> 0.960 | No clear effect |
| use AutoL2 dynamical L2 schedule instead of tuned constant L2 (no LR schedule) | WRN28-10 · CIFAR-10 | test accuracy and training speed | Improved |
| use AutoL2 together with a learning rate schedule | WRN28-10 · CIFAR-10 | test accuracy | Hurt |
| use small nonzero L2 instead of none | overparameterized networks trained to optimum · CIFAR-10 WRN28-10 | test accuracy | Improved |
Closest papers by meaning
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.89)
- Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks (NeurIPS 2019 · similarity 0.88)
- Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence (NeurIPS 2019 · similarity 0.88)
- High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks (JMLR 2026 · similarity 0.88)
- On the Weight Dynamics of Deep Normalized Networks (ICML 2024 · similarity 0.87)
- Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
