Research map / Probabilistic and scientific modeling / Bayesian optimization
Pre-trained Gaussian Processes for Bayesian Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperBO, a Bayesian optimization framework that pre-trains a GP prior on multi-task data by minimizing a KL-divergence-based loss (EKL or NLL), with theoretical bounds, and releases the PD1 multi-task hyperparameter tuning dataset.
Problem
Bayesian optimization needs a well-specified GP prior, which is hard to quantify manually for hyperparameter tuning of complex deep learning models; the paper seeks to learn this prior automatically from data on related tasks.
Approach
Pre-train a GP's mean and kernel (parameterized by neural networks) by minimizing KL divergence to the unknown ground-truth GP, using either an empirical KL (EKL) estimate from matching inputs or a negative log-likelihood (NLL) approximation; the frozen pre-trained GP is then used as the prior in standard BO acquisition functions (GP-UCB, PI), with regret bounds under unknown ground truth.
Key results
- On average HyperBO locates good hyperparameters at least 3 times more efficiently than the best competing methods on the PD1 tuning dataset.
- On HPO-B's 16 multi-task benchmarks HyperBO obtains at least 10 times speedup over competitive baselines on average.
- Theory gives bounded posterior predictions and near-zero simple regret without assuming the ground truth GP prior is known.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pre-trained GP priors for Bayesian optimization | HyperBO · PD1 deep learning optimizer tuning | BO iterations needed to obtain best validation accuracy: at least 3 times speedup | Helped |
| Use pre-trained GP priors for Bayesian optimization | HyperBO · HPO-B | BO efficiency: at least 10 times speedup | Helped |
Surprising findings
- Theory implies that less confidence should be placed in the pre-trained GP as more observations on the test function are collected, since epistemic variance can become overconfident.
Limitations
- Regret bounds are proven only for finite input domains with full observations on training functions.
- EKL can be degenerate with few matching-input tasks and needs heuristics for generic data; NLL and EKL landscapes differ.
- Pre-trained GP can diverge from ground truth with few training functions, causing harmful posteriors.
Experimental setup
Datasets: PD1 HPO-B ImageNet protein sequence dataset
Benchmarks: PD1 HPO-B
Metrics: simple regret BO iterations to best validation accuracy speedup
Baselines: random search GP-EI multi-task GP RFGP STBOH MIMO HPO-B baselines
Builds on: Gaussian process optimization GP-UCB probability of improvement multi-task learning
Closest papers by meaning
- Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.86)
- Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization (ICLR 2025 · similarity 0.86)
- Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes (NeurIPS 2024 · similarity 0.85)
- BayeSQP: Bayesian Optimization through Sequential Quadratic Programming (NeurIPS 2025 · similarity 0.85)
- BARK: A Fully Bayesian Tree Kernel for Black-box Optimization (ICML 2025 · similarity 0.85)
- Bayesian Optimization via Continual Variational Last Layer Training (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.