AI Paper Atlas

Research map / Probabilistic and scientific modeling / Bayesian optimization

Scaling Laws for Hyperparameter Optimization

Arlind Kadra, Maciej Janowski, Martin Wistuba, Josif Grabocka

NeurIPS 2023 · Probabilistic and scientific modeling › Bayesian optimization · approach: Multi-fidelity hyperparameter optimization · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use power law regressor instead of neural network per curvelearning curve forecaster · LCBenchrank correlation of final performanceImproved
condition power law on hyperparameter configuration (DPL vs PL)power law forecaster · LCBenchrank correlation of final performanceImproved
use shifted power law (Candidate 1) instead of simple power law (ablation)DPL surrogate · LCBenchregretNo clear effect
add budget scaling to power law (Candidate 2) (ablation)DPL surrogate · LCBenchregretHurt
use Broken Power Law (BPL) instead of simple power law (ablation)DPL surrogate · LCBenchregretHurt
min-smooth learning curves (ablation)DPL surrogate · LCBenchregretNo clear effect
use DPL instead of baseline HPO methodsmulti-fidelity HPO · LCBench, TaskSet, PD1normalized regret: 5 out of 6Improved
use DPL instead of baselines on PD1 at 50% budgetmulti-fidelity HPO · PD1regret rankHurt
include surrogate overhead in wall-clock timeDPL · LCBench, PD1normalized regret over timeNo clear effect
use DPL to tune learning rate hyperparameters with embedding size fidelityGPT-2 on OpenWebText · nanoGPT-Benchincumbent valueImproved
use DPL for continuous search space (learning rate, weight decay)EfficientNetV2 · CIFAR10anytime performanceImproved
increase embedding size from 6 to 384 for DPL overheadDPL · nanoGPT-Benchshare of time per iteration: 10% -> 5%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy