Research map / Architectures and efficiency / RNNs and normalization
Stepsize anything: A unified learning rate schedule for budgeted-iteration training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Unified Budget-Aware (UBA) learning rate schedule, derived from a min-max finite-iteration optimization over Hessian eigenvalue bounds, with a single hyperparameter φ, and evaluates it against common schedules on CIFAR/ImageNet ResNets and OLMo language models.
Problem
Learning rate schedules for budgeted-iteration training (fixed small iteration budgets) are heuristic, lack theoretical grounding, and require costly trial-and-error selection.
Approach
Formulates learning rate choice as a min-max finite-horizon optimization over quadratic-approximated loss curvature bounds, solves it numerically, and fits the solution with a one-parameter cosine-based closed form (UBA) whose shape parameter φ relates to the condition number.
Key results
- On CIFAR10 with ResNet18 at 100% budget, UBA reaches 95.65% vs 95.68% for budgeted training (BT) and 95.52% for cosine.
- On ImageNet ResNet50 at 25% budget UBA scores 76.29% vs 75.90% for the best baseline (BT).
- On OLMo 300M UBA reaches 50.42 average vs 49.70 for the best baseline (REX).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UBA instead of the strongest listed baseline schedule | ResNet18 on CIFAR10 · CIFAR10, 75 epochs (25%) | validation accuracy: 94.49$\pm$0.3225 -> 94.57$\pm$0.3281 | Helped |
| Use UBA instead of the strongest listed baseline schedule | ResNet18 on CIFAR10 · CIFAR10, 150 epochs (50%) | validation accuracy: 95.03$\pm$0.1353 -> 95.26$\pm$0.2150 | Helped |
| Use UBA instead of the strongest listed baseline schedule | ResNet18 on CIFAR10 · CIFAR10, 300 epochs (100%) | validation accuracy: 95.68$\pm$0.1168 -> 95.65$\pm$0.1589 | Hurt |
| Use UBA instead of the strongest listed baseline schedule | ResNet34 on CIFAR100 · CIFAR100, all listed budgets | validation accuracy | Helped |
| Use UBA instead of the strongest listed baseline schedule | ResNet50 on ImageNet · ImageNet, all listed budgets | validation accuracy | Helped |
| Use UBA instead of the strongest baseline schedule | OLMo 36M · OLMo 36M benchmarks (average) | Avg. accuracy (%): 40.92 -> 40.97 | Helped |
| Use UBA instead of the strongest baseline schedule | OLMo 73M · OLMo 73M benchmarks (average) | Avg. accuracy (%): 42.53 -> 42.73 | Helped |
| Use UBA instead of the strongest baseline schedule | OLMo 150M · OLMo 150M benchmarks (average) | Avg. accuracy (%): 45.91 -> 45.91 | No clear effect |
| Use UBA instead of the strongest baseline schedule | OLMo 300M · OLMo 300M benchmarks (average) | Avg. accuracy (%): 49.70 -> 50.42 | Helped |
| Choose smaller φ for AdamW and larger φ for SGD (ablation) | UBA schedule · Optimizer sensitivity experiments | performance | Mixed |
| Vary the number of phases K (ablation) | UBA schedule · Phase-count experiments | schedule fit to loss-surface dynamics | Helped |
| Use one shared φ across phases rather than selecting φ per phase (ablation) | Multi-phase UBA · Phase-count experiments | ease of parameter selection | Hurt |
Surprising findings
- At the 100% budget on CIFAR10 with ResNet18, the budgeted-training baseline (BT) edges out UBA in mean accuracy, and at 36M OLMo UBA's average is below the best baseline by only a small margin.
Limitations
- Results are from a single run with fixed seeds, no statistical significance for main comparisons (only curve-fit error bars).
- Explicit relationship between φ and optimization difficulty/condition number is not established; φ selection remains empirical.
- Gains are modest and not uniform; UBA is not best on all benchmarks (e.g., some OLMo 36M and 150M tasks).
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet OLMo pretraining corpus
Benchmarks: CIFAR10 CIFAR100 ImageNet PIQA HellaSwag OpenBookQA SciQ ARC-Easy ARC-Challenge COPA SIQA SocialIQA
Metrics: validation accuracy downstream accuracy average accuracy training loss validation loss
Baselines: Step decay Cosine annealing CLR OneCycle Budgeted training (linear decay) REX
Builds on: cosine annealing cyclical learning rates REX Chebyshev polynomial minimax
Closest papers by meaning
- Taking the Road Less Scheduled with Adaptive Polyak Steps (NeurIPS 2026 · similarity 0.86)
- RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks (ICLR 2025 · similarity 0.85)
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.85)
- The Road Less Scheduled (NeurIPS 2024 · similarity 0.85)
- MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search (NeurIPS 2026 · similarity 0.85)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.