Research map / Probabilistic and scientific modeling / Bayesian optimization
Informed Initialization for Bayesian Optimization and Active Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HIPE, a hyperparameter-free information-theoretic acquisition function for initial batch design that combines EPIG-style predictive entropy reduction with BALD-style hyperparameter information, weighted by EIG, and evaluates it in batch active learning and few-shot BO with fully Bayesian GPs.
Problem
Initial experimental designs for Bayesian optimization and active learning are typically space-filling or random, which neither reduces predictive uncertainty nor learns GP hyperparameters well in the few-shot, large-batch setting.
Approach
Combine an EPIG-style predictive entropy reduction term with a BALD-style hyperparameter information term, where the weight beta is set to the expected information gain of test values about hyperparameters from existing data (so no extra hyperparameter is tuned); with beta=1 it reduces to joint information gain over test values and hyperparameters, and a closed-form GP Monte Carlo estimator enable
Key results
- On batch active learning over synthetic and LCBench tasks (q=16, 4 batches, 100 seeds), HIPE is the only method consistently ranking in the top two for both RMSE and NLL.
- In the two-shot BO setting on five LCBench tasks, HIPE achieves the highest final performance on four of five tasks.
- On the 40D SVM task, HIPE infers last-two-dimension lengthscales about half an order of magnitude smaller than Sobol, though inference performance margin over next-best methods is narrow.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine predictive information gain with hyperparameter information gain | HIPE initialization · Batch active learning; synthetic and LCBench tasks | RMSE and NLL | Helped |
| use hyperparameter-focused BALD rather than balanced HIPE initialization | Fully Bayesian GP active learning · Batch active learning; synthetic and LCBench tasks | RMSE | Hurt |
| use predictive-variance-focused NIPV rather than balanced HIPE initialization | Fully Bayesian GP active learning · Batch active learning; synthetic and LCBench tasks | NLL | Hurt |
| use HIPE initialization rather than competing methods | Two-shot BO · Noisy synthetic benchmark functions | Out-of-sample inference performance | Helped |
| use HIPE initialization rather than competing methods | Two-shot BO · LCBench HPO tasks | Final performance | Helped |
| use HIPE initialization rather than Sobol initialization | Fully Bayesian GP on Feature-reduced SVM · Feature-reduced SVM (40D) | Inferred lengthscale for relevant dimensions | Helped |
| increase initialization batch size | HIPE on LCBench tasks · LCBench tasks; initialization only | NLL | Helped |
| increase initialization batch size | HIPE on LCBench tasks · LCBench tasks; initialization only | RMSE | Hurt |
| use HIPE with smaller batches rather than other initializations | Active learning · Synthetic and LCBench tasks; q=8 | NLL | Helped |
| increase active-learning batch size for HIPE | Active learning · Synthetic and LCBench tasks; q=24 | RMSE | Helped |
| use HIPE instead of NIPV | Acquisition optimization · 4-batch optimization; q=8 | Acquisition optimization runtime (sec): 19.61 $\pm$ 1.12 vs 8.62 $\pm$ 0.37 | Hurt |
Surprising findings
- On RMSE in the batch-size ablation, HIPE often underperforms simpler baselines like Sobol and Random despite leading on NLL.
- On the 20D SVM task HIPE is only mid-range in performance.
Limitations
- Nested Monte Carlo estimator makes HIPE computationally expensive for large batch sizes and high dimensions.
- Implementation is specific to GP surrogates and requires re-derivation for other Bayesian models.
- Focuses on cold-start initialization; no transfer learning or non-uniform test distributions explored.
Experimental setup
Datasets: LCBench OpenML CTSlice
Benchmarks: Ackley 4D Hartmann 4D Hartmann 6D Hartmann 12D LCBench SVM 20D SVM 40D
Metrics: RMSE NLL out-of-sample inference performance relative rank
Baselines: Sobol Random BALD NIPV LHS-Beta
Builds on: EPIG BALD NIPV qLogNEI
Closest papers by meaning
- Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.89)
- Posterior Inference with Diffusion Models for High-dimensional Black-box Optimization (ICML 2025 · similarity 0.88)
- Optimizing the Unknown: Black Box Bayesian Optimization with Energy-Based Model and Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Batched Energy-Entropy acquisition for Bayesian Optimization (NeurIPS 2024 · similarity 0.88)
- Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes (NeurIPS 2024 · similarity 0.88)
- Bayesian Optimisation with Unknown Hyperparameters: Regret Bounds Logarithmically Closer to Optimal (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.