Research map / Optimization / Implicit bias and generalization
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a large initial learning rate and anneal it later instead of using a small learning rate throughout | Two-layer ReLU network · Constructed distribution | test classification error: O(pκ log \frac{1}{\varepsilon_1}) versus Ω(p) | Helped |
| Add annealed Gaussian noise before activation layers | WideResNet16 with a small learning rate · CIFAR-10 validation set, no data augmentation | validation accuracy: 4.72% | Helped |
| Train with a small initial learning rate on patch-augmented images rather than the full dataset | WideResNet16 · Patch-augmented CIFAR-10, clean validation images | validation accuracy: 83% -> 70% | Hurt |
Closest papers by meaning
- Where Do Large Learning Rates Lead Us? (NeurIPS 2024 · similarity 0.91)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.87)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.86)
- Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate (ICLR 2024 · similarity 0.86)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.86)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.