Research map / Probabilistic and scientific modeling / Bayesian optimization
Scaling Laws for Hyperparameter Optimization
Arlind Kadra, Maciej Janowski, Martin Wistuba, Josif Grabocka
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use power law regressor instead of neural network per curve | learning curve forecaster · LCBench | rank correlation of final performance | Improved |
| condition power law on hyperparameter configuration (DPL vs PL) | power law forecaster · LCBench | rank correlation of final performance | Improved |
| use shifted power law (Candidate 1) instead of simple power law (ablation) | DPL surrogate · LCBench | regret | No clear effect |
| add budget scaling to power law (Candidate 2) (ablation) | DPL surrogate · LCBench | regret | Hurt |
| use Broken Power Law (BPL) instead of simple power law (ablation) | DPL surrogate · LCBench | regret | Hurt |
| min-smooth learning curves (ablation) | DPL surrogate · LCBench | regret | No clear effect |
| use DPL instead of baseline HPO methods | multi-fidelity HPO · LCBench, TaskSet, PD1 | normalized regret: 5 out of 6 | Improved |
| use DPL instead of baselines on PD1 at 50% budget | multi-fidelity HPO · PD1 | regret rank | Hurt |
| include surrogate overhead in wall-clock time | DPL · LCBench, PD1 | normalized regret over time | No clear effect |
| use DPL to tune learning rate hyperparameters with embedding size fidelity | GPT-2 on OpenWebText · nanoGPT-Bench | incumbent value | Improved |
| use DPL for continuous search space (learning rate, weight decay) | EfficientNetV2 · CIFAR10 | anytime performance | Improved |
| increase embedding size from 6 to 384 for DPL overhead | DPL · nanoGPT-Bench | share of time per iteration: 10% -> 5% | Improved |
Closest papers by meaning
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.89)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.87)
- Investigating the Overlooked Hessian Structure: From CNNs to LLMs (ICML 2025 · similarity 0.86)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.85)
- Principled Architecture-aware Scaling of Hyperparameters (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
