Research map / Optimization / Optimizers
Don't Decay the Learning Rate, Increase the Batch Size
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace learning-rate decay with batch-size increases | Wide ResNet 16-4 · CIFAR-10 | test accuracy: 94.3% -> 94.4% | No clear effect |
| replace learning-rate decay with batch-size increases | Wide ResNet 16-4 · CIFAR-10 | parameter updates: ~80000 -> ~29000 | Helped |
| increase initial learning rate from 0.1 to 0.5 and scale the initial batch size from 128 to 640 (ablation) | Wide ResNet 16-4 with increasing batch sizes · CIFAR-10 | parameter updates: ~29000 -> under 6500 | Helped |
| increase initial learning rate from 0.1 to 0.5 and scale the initial batch size from 128 to 640 (ablation) | Wide ResNet 16-4 with increasing batch sizes · CIFAR-10 | test accuracy: 94.4% -> 94.5% | No clear effect |
| increase momentum coefficient from 0.9 to 0.98 (ablation) | Wide ResNet 16-4 with increased initial learning rate and increasing batch sizes · CIFAR-10 | parameter updates: under 6500 -> less than 2500 | Helped |
| increase momentum coefficient from 0.9 to 0.98 (ablation) | Wide ResNet 16-4 with increased initial learning rate and increasing batch sizes · CIFAR-10 | test accuracy: 94.5% -> 93.3% | Hurt |
| replace learning-rate decay with a batch-size increase at the first step | Inception-ResNet-V2 · ImageNet validation | test accuracy: 78.7% and 77.8% -> 78.1% and 76.8% | No clear effect |
| replace learning-rate decay with batch-size increases | Inception-ResNet-V2 · ImageNet | parameter updates: just over 14000 -> below 6000 | Helped |
| increase momentum from 0.9 to 0.95 (ablation) | Inception-ResNet-V2 with increasing batch sizes · ImageNet | parameter updates: just under 6000 -> under 3500 | Helped |
| increase momentum from 0.9 to 0.95 (ablation) | Inception-ResNet-V2 with increasing batch sizes · ImageNet validation | test accuracy: 78.8% -> 78.1% and 77.8% | Hurt |
| increase momentum from 0.95 to 0.975 (ablation) | Inception-ResNet-V2 with increasing batch sizes · ImageNet | parameter updates: under 3500 -> under 2500 | Helped |
| increase batch size from 8192 to 16384 after epoch 30 rather than using 16384 throughout (ablation) | ResNet-50 · ImageNet validation | validation accuracy: 75.0% -> 76.1% | Helped |
Closest papers by meaning
- What Stops SGD on LLM Pre-Training: The Need for Large Learning Rate and How to Achieve it (NeurIPS 2026 · similarity 0.88)
- The Marginal Value of Momentum for Small Learning Rate SGD (ICLR 2024 · similarity 0.87)
- Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions (NeurIPS 2022 · similarity 0.87)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.87)
- Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling (NeurIPS 2024 · similarity 0.87)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.