Research map / Optimization / Optimizers
Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LARS with CLARS and omit warmup | ResNet50 · ImageNet, B=16384 | Top-1 testing accuracy: 75.1 -> 75.1 | No clear effect |
| Replace LARS with CLARS and omit warmup | DenseNet121 · ImageNet, B=16384 | Top-1 testing accuracy: 70.4 -> 73.6 | Helped |
| Replace LARS with CLARS and omit warmup | MobileNetv2 · ImageNet, B=16384 | Top-1 testing accuracy: 69.6 -> 70.9 | Helped |
| Replace LARS with CLARS without warmup | ResNet50, DenseNet121, and MobileNetv2 · ImageNet | training loss | Helped |
| Use CLARS instead of warmup, FC freezing, or no warmup | ResNet56 and VGG11 · CIFAR-100, B=4096 | Top-1 testing accuracy | Helped |
| Add warmup to LARS | LARS · large-batch training | convergence | Helped |
| Add CLARS layer-wise adaptive scaling instead of using no warmup (ablation) | ResNet56 and VGG11 · CIFAR-100, B=4096 | Top-1 testing accuracy | Helped |
| Freeze fully connected layers in the first few epochs (ablation) | VGG11 · CIFAR-100, B=4096 | Top-1 testing accuracy | Mixed |
| Use CLARS rather than LARS with warmup | ResNet50 · ImageNet, B=16384 | convergence speed | Helped |
Closest papers by meaning
- Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling (NeurIPS 2024 · similarity 0.86)
- How Does Critical Batch Size Scale in Pre-training? (ICLR 2025 · similarity 0.85)
- Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training (NeurIPS 2024 · similarity 0.85)
- A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks (NeurIPS 2024 · similarity 0.85)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.84)
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.