Research map / Learning theory and algorithms / Kernels and regression theory
Where Do Large Learning Rates Lead Us?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically studies how the initial learning rate (within the three-regime taxonomy) affects generalization after fine-tuning/SWA, loss-landscape geometry, and feature sparsity, finding that LRs just above the convergence threshold (subregime 2A) are optimal.
Problem
How large an initial learning rate is needed for optimal generalization, and how models trained with different initial LRs differ in loss landscape geometry and learned features.
Approach
Uses scale-invariant networks trained on a sphere with fixed LRs to isolate the effect of the initial LR; pre-trains at fixed LRs, then fine-tunes with small LRs or weight-averages, and links the optimal LR range (subregime 2A, just above the convergence threshold) to a convex basin and sparse, relevant feature learning.
Key results
- Optimal pre-training LRs lie in subregime 2A just above the convergence threshold; fine-tuned models there reach about 95% test accuracy on CIFAR-10 in the practical setting, matching a standard LR schedule.
- Pre-training in subregime 2A locates a convex basin of linearly connected high-quality minima, while small initial LRs yield unstable minima and large LRs yield diverse, unconnected minima.
- Feature sparsity peaks in subregime 2A, with the model focusing on mid-frequency features on CIFAR-10 that matter most for classification.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train with an LR in subregime 2A rather than a small regime-1 LR, then fine-tune | SI ResNet-18 · CIFAR-10 | test accuracy | Helped |
| Use a higher LR in subregime 2B instead of subregime 2A before fine-tuning or SWA | SI ResNet-18 · CIFAR-10 | test accuracy | Hurt |
| Use a low regime-1 LR for pre-training, then fine-tune with an FLR greater than the PLR | SI ResNet-18 · CIFAR-10 | test accuracy | Helped |
| Fine-tune regime-3 pre-training with a small FLR rather than train from scratch with that FLR | SI ResNet-18 · CIFAR-10 | test accuracy | Helped |
| Add an intermediate fine-tuning step through subregime 2A before the final regime-1 LR (ablation) | Pre-training in subregime 2B · CIFAR-10 | test accuracy | Helped |
| Increase the initial LR from regime 1 toward subregime 2A | SI ResNet-18 · Synthetic example | single-feature test accuracy | Helped |
| Increase the initial LR beyond subregime 2A into subregime 2B | SI ResNet-18 · Synthetic example | test accuracy | Hurt |
| Increase the initial LR from regime 1 into subregime 2A | SI ResNet-18 · CIFAR-10 frequency-band evaluation | mid-frequency accuracy | Helped |
| Increase the initial LR beyond subregime 2A into subregime 2B | SI ResNet-18 · CIFAR-10 frequency-band evaluation | mid-frequency accuracy | Hurt |
| Use subregime 2A pre-training rather than other initial LR ranges | Practical ResNet-18 · CIFAR-10 | test accuracy: ∼95% | Helped |
| Use subregime 2A pre-training rather than other initial LR ranges | Practical ResNet-18 · CIFAR-100 and Tiny ImageNet | test accuracy | Helped |
| Use subregime 2A pre-training rather than other initial LR ranges | SI ConvNet · CIFAR-10 and CIFAR-100 | test accuracy | Mixed |
Surprising findings
- Fine-tuning in the third (divergence) regime can improve over training from scratch with small LRs, due to an uneven distribution of weight norms.
- Feature sparsity is non-monotonic in initial LR, contrary to prior reports of monotonic increase with LR.
Limitations
- Purely empirical study with no theoretical support for the conclusions.
- Limited to specific datasets (CIFAR-10/100, Tiny ImageNet, synthetic) and architectures (ConvNet, ResNet-18, ViT-S/4).
- Scale-invariant setup may overlook effects of other network invariances such as rescale invariance of ReLU.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet synthetic 2D tick feature dataset
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet
Metrics: test accuracy linear-path error barrier angular distance single-feature accuracy frequency band accuracy
Baselines: fixed-LR training from scratch standard LR schedules
Builds on: three regimes of training on the sphere stochastic weight averaging catapult effect
Closest papers by meaning
- Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks (NeurIPS 2019 · similarity 0.91)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.89)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.87)
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.87)
- Initializing Models with Larger Ones (ICLR 2024 · similarity 0.86)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.