Research map / Optimization / Optimizers
Prodigy: An Expeditiously Adaptive Parameter-Free Learner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prodigy, a weighted modification of D-Adaptation that estimates the distance to solution D and improves its convergence rate by a sqrt(log) factor, with Adam-like and Dual Averaging variants evaluated on convex and deep learning tasks.
Problem
Setting learning rates automatically (learning-rate-free / parameter-free optimization) requires knowing the distance to solution D; existing D-Adaptation converges slower than the optimal sqrt-log rate in D/d0 and lags tuned Adam on some deep learning tasks.
Approach
Modify D-Adaptation's step size with AdaGrad-like denominators weighted by d_i^2 and extra weights lambda_k (e.g., lambda_k = d_k^2 for DA, or exponential weights for Adam), which removes the lambda_n-based pessimistic bound and yields a sqrt(log(D/d0)) rate matching coin-betting.
Key results
- Prodigy improves on D-Adaptation's convergence rate by a factor of sqrt(log(D/d0)), matching the coin-betting sqrt-log dependence.
- On ViT training on ImageNet, Prodigy reaches 74.63% test accuracy vs 75.4% for tuned Adam and more than 2% above D-Adaptation (weight decay 0.05).
- On convex logistic regression over 12 datasets, Prodigy outperforms D-Adaptation and adapts faster than DoG on 10 of 12 problems.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Prodigy instead of D-Adaptation | Adam-like optimization · ImageNet ViT | test accuracy: more than 2% | Helped |
| Use Prodigy instead of tuned Adam | ViT · ImageNet ViT | test accuracy: 74.63% compared to 75.4% | Hurt |
| Set weight decay to 0.05 instead of 0.1 (ablation) | all optimizer variants · ImageNet ViT | test accuracy | Helped |
| Use Prodigy instead of D-Adapted Adam | VGG11 and ResNet-50 · CIFAR10 | test accuracy | Helped |
| Use Prodigy instead of D-Adapted Adam | LSTM · IWSLT14 | training loss | Helped |
| Use Prodigy instead of the baselines | LSTM · IWSLT14 | test loss | Hurt |
| Use Prodigy instead of the baseline | RoBERTa and GPT · BookWiki | performance | No clear effect |
| Use Prodigy instead of the baselines | DLRM · Criteo Kaggle Display Advertising | performance | No clear effect |
| Use Prodigy instead of the baselines | VarNet · fastMRI Knee MRI | performance | No clear effect |
| Use Prodigy instead of tuned AdamW | nanoGPT · Shakespeare | training loss | No clear effect |
| Use Prodigy instead of D-Adapted Adam | nanoGPT · Shakespeare | training loss | Helped |
| Use DoG or L-DoG instead of Prodigy | nanoGPT · Shakespeare | test loss | Hurt |
Surprising findings
- D-Adaptation significantly underperforms tuned Adam on ViT training, a gap that persists across weight-decay values.
- The estimated step size in Prodigy was very consistent across 8 random seeds on nanoGPT.
Experimental setup
Datasets: CIFAR10 ImageNet IWSLT14 Criteo Kaggle Display Advertising BookWiki Shakespeare fastMRI knee MRI 12 logistic-regression benchmark datasets
Benchmarks: CIFAR10 ImageNet IWSLT14 Criteo Kaggle BookWiki fastMRI knee MRI Shakespeare
Metrics: test accuracy test loss train loss functional gap
Baselines: D-Adaptation Adam AdamW DoG L-DoG Adam with tuned learning rate
Builds on: D-Adaptation AdaGrad Adam DoG
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.88)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.88)
- Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad (NeurIPS 2024 · similarity 0.88)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.87)
- AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretraining (NeurIPS 2025 · similarity 0.87)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.