AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule

Nikhil Iyer, V. Thejas, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu

JMLR 2023 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add a longer high-LR explore phase (10 -> 100 epochs at LR 0.1) (ablation)step LR schedule · CIFAR-10 sharpnessKeskar sharpness: 10.56 -> 3.54Improved
add a longer high-LR explore phase (0 -> 100 epochs) (ablation)step LR schedule · CIFAR-10 Fisher scoreFIM score: 0.051 -> 0.042Improved
add more finer-scale explore at LR 0.01 (10 -> 50 epochs) (ablation)step LR schedule · CIFAR-10 (200 epochs)test accuracy: 94.78 -> 95.10Improved
train 50x longer (10,000 epochs) at fixed low LR 0.001 instead of exploring at high LR (ablation)ResNet-18 · CIFAR-10test accuracy: 93.9 vs over 95Hurt
add longer explore phase to Knee schedule (5 -> 40 epochs) (ablation)Knee schedule · IWSLT'14 DE-EN (50 epochs)BLEU: 34.93 -> 35.53Improved
fix explore to 50% instead of tuning explore duration (ablation)Knee schedule · ImageNet (90 epochs)top-1 accuracy: 76.71 -> 76.58No clear effect
replace step schedule with Knee schedule at original budgetResNet-50 · ImageNet (90 epochs)top-1 accuracy: 75.87 -> 76.71Improved
replace linear decay with Knee schedule at 2/3 BERT-large budgetBERT-large pretraining · SQuAD v1.1 F1F1: 90.64 -> 91.29Improved
replace linear decay with Knee schedule on fine-tuningBERT-base · SQuAD v1.1 fine-tuningEM: 80.89 -> 81.38Improved
replace inverse sqrt schedule with Knee scheduleCutoff Transformer · IWSLT'14 DE-ENBLEU: 37.6 -> 37.78Improved
replace One-Cycle with Knee scheduleResNet-18 · CIFAR-10 (200 epochs)accuracy: 94.09 -> 95.26Improved
replace cosine decay with Knee schedule at long budgetResNet-18 · CIFAR-10 (200 epochs)accuracy: 95.23 -> 95.26No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy