Research map / Probabilistic and scientific modeling / Bayesian optimization
Efficient Hyperparameter Optimization with Adaptive Fidelity Identification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FastBO, a multi-fidelity model-based HPO method that adaptively picks each configuration's fidelity via learning-curve-derived efficient and saturation points, and shows it can extend single-fidelity methods to the multi-fidelity setting.
Problem
Bayesian optimization for hyperparameter optimization and NAS is costly because full evaluations are needed; multi-fidelity extensions struggle to choose an appropriate fidelity for each configuration to fit the surrogate model.
Approach
Define an efficient point (where doubling resources yields improvement below a threshold) as each configuration's surrogate-fitting fidelity, and a saturation point as its approximate final fidelity, both obtained by fitting a weighted combination of power-law, exponential and logarithmic learning curve models to early observations; add warm-up early termination and post-processing promotion to sa
Key results
- FastBO saves 10% to 87% wall-clock time over other methods on configuration identification (Table 2), with 3-8x speedup over vanilla BO.
- FastBO achieves top-2 final performance on 8 of 10 datasets.
- Multi-fidelity variants FastCQR, FastBORE and FastREA outperform their single-fidelity counterparts on three datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive efficient-point fidelity instead of a fixed fidelity (ablation) | FastBO · Slice | anytime validation loss | Helped |
| use adaptive efficient-point fidelity instead of a fixed fidelity (ablation) | FastBO · ImageNet16-120 | anytime validation error | Helped |
| use adaptive efficient-point fidelity instead of a fixed fidelity (ablation) | FastBO · Covertype | anytime validation error | Helped |
| extend CQR to multi-fidelity using adaptive efficient-point evaluations (ablation) | CQR · Slice, ImageNet16-120, and Covertype | anytime performance | Helped |
| extend BORE to multi-fidelity using adaptive efficient-point evaluations (ablation) | BORE · Slice, ImageNet16-120, and Covertype | anytime performance | Helped |
| extend REA to multi-fidelity using adaptive efficient-point evaluations (ablation) | REA · Slice, ImageNet16-120, and Covertype | anytime performance | Helped |
| use FastBO instead of BO | HPO configuration identification · Covertype | wall-clock time: 2.9±0.7 -> 0.7±0.3 | Helped |
| use FastBO instead of Hyper-Tune | HPO configuration identification · ImageNet16-120 | wall-clock time: 3.4±1.1 -> 2.2±0.7 | Helped |
| use FastBO instead of A-BOHB | HPO configuration identification · Slice | wall-clock time: 2.1±0.7 -> 0.4±0.1 | Helped |
Surprising findings
- Fixed partial-evaluation fidelities (25%, 50%, 75%) converge faster early but end up with suboptimal final performance compared to vanilla BO.
Limitations
- Final performance is not the best because evaluations are capped at saturation points.
- Relies on predefined thresholds (delta1, delta2, alpha) and parametric learning curve model choice.
Experimental setup
Datasets: LCBench NAS-Bench-201 FCNet NAS-Bench-301
Benchmarks: LCBench NAS-Bench-201 FCNet NAS-Bench-301
Metrics: validation accuracy/error/loss over wall-clock time wall-clock time to identify a satisfactory configuration relative efficiency
Baselines: Random search Gaussian process BO ASHA Hyperband PASHA A-BOHB A-CQR BOHB DyHPO Hyper-Tune
Builds on: BOHB ASHA Hyperband Successive halving PASHA
Closest papers by meaning
- BOHB: Robust and Efficient Hyperparameter Optimization at Scale (ICML 2018 · similarity 0.89)
- Multi-Fidelity Bayesian Optimization via Deep Neural Networks (NeurIPS 2020 · similarity 0.89)
- Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks (NeurIPS 2021 · similarity 0.89)
- Bayesian Optimisation with Unknown Hyperparameters: Regret Bounds Logarithmically Closer to Optimal (NeurIPS 2024 · similarity 0.88)
- Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning (NeurIPS 2025 · similarity 0.87)
- Informed Initialization for Bayesian Optimization and Active Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.