Research map / Probabilistic and scientific modeling / Bayesian optimization
Bayesian Optimization via Continual Variational Last Layer Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VBLL-based surrogate models for Bayesian optimization with an online continual training loop that interleaves recursive last-layer conditioning and periodic full feature retraining.
Problem
Gaussian processes are the standard BO surrogates but struggle with complex, non-Euclidean, high-dimensional, and non-stationary correlation structures and need kernel design; existing BNN surrogates often underfit and are hard to train and condition online.
Approach
Shows that the optimum of the VBLL variational lower bound equals the exact Bayesian linear regression posterior (over the full data), which allows interleaving mini-batch full-model training with cheap rank-1 Cholesky recursive last-layer updates; also uses Thompson sampling analytically over the parametric last layer.
Key results
- VBLLs with Thompson sampling are the most consistently strong surrogates on high-dimensional, non-stationary benchmarks (NNdraw 200D, Pestcontrol 25D, Lunarlander 12D), where GPs often underperform.
- On classic low-dimensional benchmarks VBLLs perform on par with or better than most BNN surrogates and match well-tuned GPs; on Ackley2D and Ackley5D with TS they surpass GPs.
- Continual VBLL variant (event-triggered re-initialization with recursive last-layer updates) substantially reduces surrogate fit time while keeping performance close to standard VBLLs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VBLL instead of GP surrogates | Bayesian optimization · High-dimensional and non-stationary benchmarks | optimization performance | Helped |
| Use VBLL instead of GP surrogates | Bayesian optimization · Classic benchmarks | optimization performance | No clear effect |
| Use VBLL with analytic Thompson sampling | Bayesian optimization · Ackley2D and Ackley5D | optimization performance | Helped |
| Use VBLL with Thompson sampling instead of other surrogates with Thompson sampling | Bayesian optimization · High-dimensional and non-stationary benchmarks | optimization performance | Helped |
| Use VBLL with Thompson sampling instead of logEHVI | Multi-objective Bayesian optimization · OilSorbent | hypervolume | Helped |
| Use event-triggered continual training with recursive last-layer updates instead of training VBLL features from scratch each iteration (ablation) | VBLL · Classic benchmarks | optimization performance | Hurt |
| Use event-triggered continual training with recursive last-layer updates instead of training VBLL features from scratch each iteration (ablation) | VBLL · High-dimensional and non-stationary benchmarks | optimization performance | No clear effect |
| Use event-triggered retraining instead of periodic retraining (M=5) (ablation) | Continual VBLL · Selected benchmarks | optimization performance | Helped |
| Use event-triggered retraining instead of sigmoid-scheduled retraining (ablation) | Continual VBLL · NNdraw with logEI | optimization performance | Hurt |
| Add early stopping based on training loss (ablation) | VBLL training · Bayesian optimization benchmarks | runtime | Helped |
| Train for a small fixed number of epochs instead of using early stopping (ablation) | VBLL training · Bayesian optimization benchmarks | optimization performance | Hurt |
| Reuse feature weights from the previous full training iteration (ablation) | Continual VBLL · Bayesian optimization benchmarks | runtime | Helped |
Surprising findings
- A deterministic MAP neural network Thompson sample baseline performs surprisingly well on several benchmarks, though with high variance.
- Event-triggered continual learning stagnates on NNdraw with logEI, while scheduled re-initialization does not.
Limitations
- Continual learning with event-triggered re-initialization can stagnate on some high-dimensional problems with logEI.
- VBLLs are the most expensive surrogate to train on smaller problems when retraining from scratch every iteration.
- Results depend on hyperparameters such as the Wishart scale, which are not well tuned for all problems (e.g., NNdraw).
Experimental setup
Datasets: Branin Ackley2D Ackley5D Hartmann NNdraw Pestcontrol Lunarlander BraninCurrin DTLZ1 DTLZ2 OilSorbent
Benchmarks: Branin Ackley2D Ackley5D Hartmann NNdraw 200D Pestcontrol 25D Lunarlander 12D BraninCurrin DTLZ1 DTLZ2 OilSorbent
Metrics: best observed value hypervolume log hypervolume difference accumulated surrogate fit time
Baselines: Gaussian process (Matern 5/2) infinite-width BNN deep kernel learning last layer Laplace approximation deep ensembles D-scaled GP
Builds on: variational Bayesian last layer deep kernel learning last layer Laplace approximation infinite-width Bayesian neural networks
Closest papers by meaning
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.91)
- Variational Bayesian Last Layers (ICLR 2024 · similarity 0.90)
- Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes (NeurIPS 2024 · similarity 0.88)
- Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.87)
- Training Bayesian Neural Networks with Sparse Subspace Variational Inference (ICLR 2024 · similarity 0.87)
- Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.