Research map / Probabilistic and scientific modeling / Bayesian optimization
A Study of Bayesian Neural Network Surrogates for Bayesian Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically benchmarks Bayesian neural network surrogates (HMC, SGHMC, LLA, deep ensembles, infinite-width BNNs, deep kernel learning) against standard Matérn GPs for Bayesian optimization across synthetic, real-world, and high-dimensional problems.
Problem
Standard Gaussian process surrogates are almost exclusively used in Bayesian optimization, but it is unclear whether Bayesian neural network surrogates with various approximate inference methods would be better alternatives, especially for non-stationary, high-dimensional, discrete, or multi-objecti
Approach
Treats the surrogate model as the object of study: evaluates a broad family of neural-network-based surrogates (finite-width BNNs under different approximate inference, infinite-width BNNs as GPs with NN kernels, LLA, deep kernel learning) against standard GPs with a common Monte-Carlo EI acquisition function, without proposing a new surrogate.
Key results
- HMC is the most successful approximate inference method for fully stochastic finite-width BNNs, comparable to GPs on synthetic benchmarks and outperforming GPs on 10-dimensional Ackley.
- Deep ensembles consistently underperform other surrogates, plateauing at lower objective values on problems like BraninCurrin and DTLZ1.
- Infinite-width BNNs (I-BNNs) consistently outperform other surrogates, including GPs, in high-dimensional problems.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HMC instead of SGHMC, linearized Laplace, or deep ensembles for finite-width BNN inference | Finite-width BNN surrogates · Synthetic and real-world Bayesian optimization benchmarks | maximum reward | Helped |
| Use deep ensembles instead of other surrogate models | Bayesian optimization surrogates · Synthetic benchmarks | objective value | Hurt |
| Use deep kernel learning instead of a standard GP | Bayesian optimization surrogates · Branin and BraninCurrin | optimization performance | No clear effect |
| Use deep kernel learning instead of a standard GP | Bayesian optimization surrogates · Ackley | optimization performance | Hurt |
| Use HMC instead of a standard GP | Finite-width BNN surrogates · 10-dimensional Ackley | maximum reward | Helped |
| Use an infinite-width BNN instead of other surrogate models | Bayesian optimization surrogates · High-dimensional problems | maximum reward | Helped |
| Use a BNN surrogate instead of a GP surrogate | Bayesian optimization surrogates · Pest Control | maximum reward | Helped |
| Use a GP surrogate instead of BNN surrogates | Bayesian optimization surrogates · Cell Coverage and Lunar Lander | maximum reward | Helped |
| Use a problem-tailored HMC network architecture rather than the default architecture (ablation) | HMC BNN surrogate · Pest Control | average maximum value | Helped |
| Increase the number of function queries (ablation) | Deep ensembles · Bayesian optimization tasks | optimization performance | Helped |
| Increase the number of function queries (ablation) | Infinite-width BNN surrogates · Bayesian optimization tasks | optimization performance | No clear effect |
| Increase the number of function queries (ablation) | Finite-width BNN surrogates · Bayesian optimization tasks | optimization performance | Helped |
Surprising findings
- Deep ensembles perform surprisingly poorly for Bayesian optimization despite their strong results in other settings.
- Deep kernel learning, which is not stochastic over network weights, is relatively competitive with fully stochastic BNNs.
Limitations
- No single surrogate dominates across problems, so results are problem-dependent.
- Full-batch HMC is computationally expensive and limited to models with a few hundred thousand parameters.
Experimental setup
Datasets: Branin Hartmann Ackley BraninCurrin DTLZ1 DTLZ5 Interferometer Lunar Lander Cellular Coverage Oil Spill Sorbent Pest Control PDE Optimization
Benchmarks: Branin Hartmann Ackley BraninCurrin DTLZ1 DTLZ5 Interferometer Lunar Lander Cell Coverage Oil Spill Sorbent Pest Control PDE Optimization
Metrics: maximum objective value found hypervolume model ranking score
Baselines: Matérn-5/2 GP RBF GP SGHMC BNN LLA deep ensembles DKL infinite-width BNN (I-BNN)
Builds on: Monte-Carlo acquisition functions deep kernel learning linearized Laplace approximation SGHMC deep ensembles
Closest papers by meaning
- Bayesian Optimization via Continual Variational Last Layer Training (ICLR 2025 · similarity 0.91)
- Modulating Surrogates for Bayesian Optimization (ICML 2020 · similarity 0.91)
- Bayesian Optimization with Robust Bayesian Neural Networks (NeurIPS 2016 · similarity 0.90)
- Training Bayesian Neural Networks with Sparse Subspace Variational Inference (ICLR 2024 · similarity 0.89)
- Optimization Proxies using Limited Labeled Data and Training Time -- A Semi-Supervised Bayesian Neural Network Approach (ICML 2025 · similarity 0.88)
- Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.