Research map / Architectures and efficiency / RNNs and normalization
Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
Nikhil Iyer, V. Thejas, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a longer high-LR explore phase (10 -> 100 epochs at LR 0.1) (ablation) | step LR schedule · CIFAR-10 sharpness | Keskar sharpness: 10.56 -> 3.54 | Improved |
| add a longer high-LR explore phase (0 -> 100 epochs) (ablation) | step LR schedule · CIFAR-10 Fisher score | FIM score: 0.051 -> 0.042 | Improved |
| add more finer-scale explore at LR 0.01 (10 -> 50 epochs) (ablation) | step LR schedule · CIFAR-10 (200 epochs) | test accuracy: 94.78 -> 95.10 | Improved |
| train 50x longer (10,000 epochs) at fixed low LR 0.001 instead of exploring at high LR (ablation) | ResNet-18 · CIFAR-10 | test accuracy: 93.9 vs over 95 | Hurt |
| add longer explore phase to Knee schedule (5 -> 40 epochs) (ablation) | Knee schedule · IWSLT'14 DE-EN (50 epochs) | BLEU: 34.93 -> 35.53 | Improved |
| fix explore to 50% instead of tuning explore duration (ablation) | Knee schedule · ImageNet (90 epochs) | top-1 accuracy: 76.71 -> 76.58 | No clear effect |
| replace step schedule with Knee schedule at original budget | ResNet-50 · ImageNet (90 epochs) | top-1 accuracy: 75.87 -> 76.71 | Improved |
| replace linear decay with Knee schedule at 2/3 BERT-large budget | BERT-large pretraining · SQuAD v1.1 F1 | F1: 90.64 -> 91.29 | Improved |
| replace linear decay with Knee schedule on fine-tuning | BERT-base · SQuAD v1.1 fine-tuning | EM: 80.89 -> 81.38 | Improved |
| replace inverse sqrt schedule with Knee schedule | Cutoff Transformer · IWSLT'14 DE-EN | BLEU: 37.6 -> 37.78 | Improved |
| replace One-Cycle with Knee schedule | ResNet-18 · CIFAR-10 (200 epochs) | accuracy: 94.09 -> 95.26 | Improved |
| replace cosine decay with Knee schedule at long budget | ResNet-18 · CIFAR-10 (200 epochs) | accuracy: 95.23 -> 95.26 | No clear effect |
Closest papers by meaning
- Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes (CVPR 2025 · similarity 0.87)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.86)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.86)
- Avoiding spurious sharpness minimization broadens applicability of SAM (ICML 2025 · similarity 0.86)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.86)
- Where Do Large Learning Rates Lead Us? (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
