Research map / Architectures and efficiency / RNNs and normalization
An Exponential Learning Rate Schedule for Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace step decay with TEXP, including momentum correction at phase transitions | PreResNet32 · CIFAR10 | test accuracy | No clear effect |
| use a cosine-based exponential learning-rate schedule instead of step decay | PreResNet32 · CIFAR10 | test accuracy | Helped |
| remove the instant learning-rate decay at phase transitions (ablation) | TEXP schedule for PreResNet32 · CIFAR10 | test accuracy | Hurt |
| add instant learning-rate decay at phase transitions (ablation) | exponential learning-rate schedule for PreResNet32 · CIFAR10 | ability to remain near a good local minimum | Mixed |
| remove weight decay while retaining normalization (ablation) | toy Gaussian-input logistic-regression setting · toy binary classification | training error | Helped |
| remove normalization while retaining weight decay (ablation) | toy Gaussian-input logistic-regression setting · toy binary classification | training error | Helped |
| combine weight decay and batch normalization (ablation) | toy Gaussian-input logistic-regression setting · toy binary classification | training error | Hurt |
Closest papers by meaning
- Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate (NeurIPS 2020 · similarity 0.89)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.89)
- Normalization and effective learning rates in reinforcement learning (NeurIPS 2024 · similarity 0.88)
- Theoretical Analysis of Auto Rate-Tuning by Batch Normalization (ICLR 2019 · similarity 0.88)
- On the Weight Dynamics of Deep Normalized Networks (ICML 2024 · similarity 0.85)
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.