Research map / Probabilistic and scientific modeling / Bayesian optimization

UQ-Guided Hyperparameter Optimization for Iterative Learners

NeurIPS 2024 · Probabilistic and scientific modeling › Bayesian optimization · approach: Multi-fidelity hyperparameter optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a UQ-guided scheme that models per-candidate learning curves with a weighted least-squares momentum-plus-bias model to estimate converged loss distributions, and uses them to choose how many candidates to keep and how to allocate budget in SH, HB, BOHB and SS.

Problem

Hyperparameter optimization for iterative learners discards candidates that look poor early in training because it ignores model training uncertainty, leading to regret in final validation loss.

Approach

Quantify model uncertainty on the fly via a weighted least-squares loss-curve model (momentum term plus latent candidate bias) to get a Gaussian estimate of each candidate's converged loss, then compare candidates by probability of being better and choose the number of survivors per round via a confidence curve balancing exploration (keeping candidates) and exploitation (more budget per candidate)

Key results

• UQ-guided methods reduce regret by 21-55% over the original HPO methods at the same exploration cost on NAS-Bench-201 and LCBench.
• The enhanced methods need only 30-75% of the time to reach accuracy comparable to the original methods (e.g., SH+ matches SH's 5% regret with about 50.78% of the budget on NAS-Bench-201).
• On ridge regression (Million Song Dataset), SH+ gives over 40% improvement over SH on average.

Measured effects reported in the paper

Surprising findings

• Successive Halving discarded the actually best candidates in the first 8-22 iterations in 100 experiments, causing 48% regret in validation loss.
• Gradient-based HPO methods may not benefit from the UQ-guided scheme because of their sequential properties.

Limitations

• Mostly suitable for iterative learners that allow ranking multiple learners during HPO; needs adaptation for other learners.
• Gradient-based HPO methods may not benefit from the scheme.
• Theoretical bounds do not prove the UQ method always outperforms the UQ-oblivious counterpart; the empirical comparison is the main evidence.

Experimental setup

Datasets: NAS-Bench-201 (CIFAR-10, CIFAR-100, ImageNet-16-120) LCBench (Fashion-MNIST and other OpenML tasks) Million Song Dataset year prediction

Benchmarks: NAS-Bench-201 LCBench Million Song Dataset year prediction (ridge regression)

Metrics: regret top-1 rank fraction of budget to reach comparable accuracy

Baselines: Successive Halving Hyperband BOHB Sub-sampling HyperJump Freeze-thaw Bayesian optimization

Builds on: Successive Halving Hyperband BOHB Sub-sampling

Closest papers by meaning

• Meta-Surrogate Benchmarking for Hyperparameter Optimization (NeurIPS 2019 · similarity 0.87)
• LAMDA: Two-Phase HPO via Learning Prior from Low-Fidelity Data (AAAI 2026 · similarity 0.87)
• Towards Learning Universal Hyperparameter Optimizers with Transformers (NeurIPS 2022 · similarity 0.87)
• Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning (AAAI 2024 · similarity 0.86)
• Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning (NeurIPS 2025 · similarity 0.85)
• Efficient Hyperparameter Optimization with Adaptive Fidelity Identification (CVPR 2024 · similarity 0.84)