Research map / Optimization / Optimizers
AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaLRS, an online learning-rate search that scales the base LR up or down by monitoring loss-descent velocity (least-squares slope) within a single pretraining run, with convergence and geometric-decay analysis.
Problem
Finding a good learning rate for foundation model pretraining is costly, since existing transfer and scaling-law approaches are scenario-specific or need proxy-model sweeps, and automatic HPO needs many independent runs.
Approach
Treat the loss descent velocity and the training loss as near-convex in the learning rate with a shared optimum, then adjust the LR online by trial upscaling and keeping or reverting it based on whether the loss slope improves, with backtracking on downscaling.
Key results
- On 2B LLM with a small initial LR, AdaLRS reduces training loss from 3.50 to 2.56 versus the small-LR cosine baseline and reaches PPL 12.43 test vs 32.66 baseline.
- On 7B LLM with a small initial LR, AdaLRS achieves test PPL 10.74 vs 15.32 for the small-LR baseline and roughly matches the fit-LR baseline (10.69).
- On 2B VLM, starting from a small LR, AdaLRS raises the average benchmark score from 53.77 to 57.34, above the fit-LR average of 56.16 in the reported table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Enable AdaLRS with an appropriate initial learning rate | Qwen2.5-1.5B pretraining · SlimPajama validation | PPL: 13.51 -> 12.42 | Helped |
| Enable AdaLRS with a large initial learning rate | Qwen2.5-1.5B pretraining · SlimPajama validation | PPL: 183.94 -> 204.97 | Hurt |
| Enable AdaLRS with a small initial learning rate | Qwen2.5-1.5B pretraining · SlimPajama validation | PPL: 12.66 -> 32.69 | Hurt |
| Enable AdaLRS with an appropriate initial learning rate | Qwen2.5-7B pretraining · SlimPajama validation | PPL: 10.67 -> 10.72 | Hurt |
| Enable AdaLRS with a large initial learning rate | Qwen2.5-7B pretraining · SlimPajama validation | PPL: 158.22 -> 165.70 | Hurt |
| Enable AdaLRS with a small initial learning rate | Qwen2.5-7B pretraining · SlimPajama validation | PPL: 10.61 -> 15.30 | Hurt |
| Enable AdaLRS from an appropriate initial learning rate | 2B VLM pretraining · 2B VLM evaluation | Average: 55.80 -> 56.16 | Helped |
| Enable AdaLRS from a large initial learning rate | 2B VLM pretraining · 2B VLM evaluation | Average: 47.67 -> 48.96 | Helped |
| Enable AdaLRS from a small initial learning rate | 2B VLM pretraining · 2B VLM evaluation | Average: 53.77 -> 57.34 | Helped |
| Add backtracking state restoration during LR downscaling (ablation) | AdaLRS in 2B LLM pretraining · 2B LLM pretraining | Training loss | Helped |
| Enable AdaLRS from a small initial learning rate | 2B VLM continual pretraining · VLM continual pretraining | Training loss: 0.8851 -> 0.8286 | Helped |
| Enable AdaLRS from a large initial learning rate | 2B VLM continual pretraining · VLM continual pretraining | Training loss | Hurt |
Surprising findings
- AdaLRS starting from a small LR surpasses the fit-LR baseline on the 7B LLM and the 2B VLM, which the authors attribute to the adjustment reaching a better LR than the grid-searched fit value.
- AdaLRS fails to recover from excessively large initial learning rates; training loss stays high, and in continual pretraining the large-LR run still shows loss above 1.88.
Limitations
- Fails to match appropriate-LR baselines when the initial learning rate is excessively large; recommends small initial LRs.
- Does not study how close AdaLRS gets to the true optimal LR; fine-grained convergence left for future work.
- Hypotheses of near-convexity and shared optimum are assumed and supported only by small-scale experiments; the continual pretraining large-LR case cannot be recovered.
Experimental setup
Datasets: SlimPajama Infinity-Instruct Infinity-MM Detail caption data OCR datasets
Benchmarks: SlimPajama validation and test PPL Alpaca-Gen KNIGHT-Gen LLaVABench MMVet MMStar DocVQA OCRBench TextVQA DetailCaps-4870
Metrics: training loss perplexity benchmark score final learning rate
Baselines: Fit LR cosine baseline Large LR cosine baseline Small LR cosine baseline Hypergradient Descent
Builds on: Tensor Programs / muP Hypergradient Descent cosine learning rate schedule WSD scheduler
Closest papers by meaning
- A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules (ICLR 2025 · similarity 0.87)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.87)
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.86)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.86)
- Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent (ICML 2025 · similarity 0.86)
- Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.