Research map / Probabilistic and scientific modeling / Bayesian optimization
Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes cost-sensitive freeze-thaw Bayesian optimization (CFBO), which maximizes a user-specified cost-performance utility, stops HPO near maximum utility via an adaptive regret-based criterion, and trains a PFN surrogate with LC mixup transfer learning.
Problem
Hyperparameter optimization under user-defined trade-offs between computational cost and performance, where standard multi-fidelity HPO assumes a large budget and maximizes final validation performance, and early stopping must be accurate from early epochs.
Approach
Introduces a user utility over (budget, best-so-far performance) estimated from pairwise preferences via Bradley-Terry, an EI-of-utility acquisition that chooses the best future target epoch, an adaptive regret-based stopping threshold based on probability of improvement, and LC-mixup transfer learning for a PFN-based LC extrapolator.
Key results
- CFBO outperforms all non-transfer and transfer baselines in most cost-sensitive settings (alpha>0) across LCBench, TaskSet and PD1 in normalized regret and rank.
- On PD1 ablation, the full CFBO (adaptive threshold + acquisition + LC mixup) reduces normalized regret from 15.2 to 1.7 (x100 scale) at alpha=2^-2 relative to the ifBO-style baseline.
- Wall-clock per BO step is 1.52s on LCBench for CFBO vs 0.58s for ifBO, with differences negligible relative to network training time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add transfer learning with LC mixup (ablation) | CFBO LC extrapolator · PD1 | normalized regret (×100): 15.2{ $\pm$2.0} -> 11.7{ $\pm$0.4} | Helped |
| use the proposed utility-based acquisition instead of ifBO acquisition (ablation) | CFBO · PD1 | normalized regret (×100): 11.7{ $\pm$0.4} -> 8.5{ $\pm$0.0} | Helped |
| add the adaptive threshold stopping criterion (ablation) | CFBO · PD1 | normalized regret (×100): 8.5{ $\pm$0.0} -> 1.7{ $\pm$0.0} | Helped |
| use CFBO instead of transfer-BO baselines | cost-sensitive HPO · TaskSet, c=1, α=2^{-2} | normalized regret | Hurt |
| use CFBO instead of transfer-BO baselines | cost-sensitive HPO · LCBench, TaskSet, and PD1 | wall-clock time per BO step: 0.58{ $\pm$0.01} vs 1.52{ $\pm$0.02} (LCBench); 0.30{ $\pm$0.00} vs 0.78{ $\pm$0.01} (TaskSet); 0.08{ $\pm$0.00} vs 0.23 | Hurt |
Limitations
- Pool-based HPO only; acquisition is not differentiable so continuous extension is not straightforward.
- Assumes users care only about the trade-off within a single task, ignoring future budgets or parallel multi-task costs.
- Stopping is weaker for weak penalties (alpha in {2^-6, 2^-5}), where CFBO struggles to stop near maximum utility.
Experimental setup
Datasets: LCBench TaskSet PD1
Benchmarks: LCBench TaskSet PD1
Metrics: normalized regret average rank wall-clock time per BO step
Baselines: Random Search BOHB DEHB DyHPO DPL ifBO Quick-Tune FSBO
Builds on: ifBO Freeze-thaw BO PFNs Quick-Tune DyHPO
Closest papers by meaning
- Bayesian Optimisation with Unknown Hyperparameters: Regret Bounds Logarithmically Closer to Optimal (NeurIPS 2024 · similarity 0.88)
- BONSAI: Bayesian Optimization with Natural Simplicity and Interpretability (NeurIPS 2026 · similarity 0.87)
- Efficient Hyperparameter Optimization with Adaptive Fidelity Identification (CVPR 2024 · similarity 0.87)
- This Too Shall Pass: Removing Stale Observations in Dynamic Bayesian Optimization (NeurIPS 2024 · similarity 0.86)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.86)
- Informed Initialization for Bayesian Optimization and Active Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.