Research map / Probabilistic and scientific modeling / Bayesian optimization
Approximation-Aware Bayesian Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Modifies sparse variational GPs for Bayesian optimization by jointly optimizing the SVGP approximation and the acquisition decision through a utility-calibrated expected utility lower bound (EULBO) for EI and KG, including batch variants.
Problem
Sparse variational GP approximations used in large-budget high-dimensional Bayesian optimization are fit to maximize global ELBO fidelity, which is misaligned with the decision goal and yields suboptimal acquisitions.
Approach
Couple SVGP fitting and acquisition selection in one objective: the ELBO plus the expected log-utility of the acquisition (utility-calibrated variational inference), giving EULBO for EI (via softplus-smoothed utility) and KG (one-shot with online Cholesky mean updates), and q-EULBO for batch BO.
Key results
- EULBO-SVGP almost always dominates ELBO-SVGP across the benchmark tasks and often needs a small fraction of the oracle calls to reach comparable performance.
- EULBO with TuRBO outperforms the other TuRBO-based baselines on high-dimensional tasks, averaged over 20 runs with standard error bars.
- EULBO adds negligible cost to KG: the Cholesky-based online update is O(m^2) per sample, amortized by the O(m^3) ELBO cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EULBO-SVGP instead of ELBO-SVGP | SVGP Bayesian optimization · High-dimensional benchmark tasks | best-found reward / sample efficiency | Helped |
| Use EULBO with TuRBO instead of ELBO-based baselines with TuRBO | TuRBO · High-dimensional benchmark tasks | best-found reward | Helped |
| Use EULBO with standard BO instead of ELBO-based baselines with standard BO | Standard BO · High-dimensional benchmark tasks | best-found reward | Helped |
| Use EULBO EI instead of ELBO EI | SVGP with TuRBO · Lasso DNA | wall-clock run time (minutes): 184.40 $\pm$ 0.59 -> 267.30 $\pm$ 2.53 | Hurt |
| Use EULBO KG instead of EULBO EI | SVGP with TuRBO · Lasso DNA | wall-clock run time (minutes): 267.30 $\pm$ 2.53 -> 296.95 $\pm$ 1.31 | Hurt |
| Increase inducing points from 100 to 1024 (ablation) | EULBO-SVGP and ELBO-SVGP with TuRBO · High-dimensional benchmark tasks | optimization performance | No clear effect |
| Use EULBO-SVGP instead of ELBO-SVGP (ablation) | TuRBO with either 100 or 1024 inducing points · High-dimensional benchmark tasks | optimization performance | Helped |
| Use EULBO-EI instead of ELBO-EI (ablation) | TuRBO with PPGPR or standard SVGP objective · High-dimensional benchmark tasks | optimization performance | Helped |
| Replace the standard SVGP objective with PPGPR (ablation) | TuRBO · High-dimensional benchmark tasks | optimization performance | No clear effect |
| Refine only variational parameters instead of all SVGP parameters with EULBO (ablation) | EULBO-refined SVGP · Lasso DNA | optimization performance | No clear effect |
| Refine all SVGP parameters instead of only variational parameters with EULBO (ablation) | EULBO-refined SVGP · Median Molecules 2 | optimization performance | Helped |
| Apply EULBO refinement to a subset of SVGP parameters after ELBO fitting (ablation) | ELBO-SVGP · Benchmark tasks | optimization performance | Helped |
Surprising findings
- Neither EI-based nor KG-based EULBO consistently outperforms the other across tasks.
Limitations
- Increased computational cost relative to standard SVGP due to warm-start and joint optimization.
- EULBO optimization requires tricks such as gradient clipping and block-coordinate updates; it is non-trivially unstable.
- Utility must be strictly positive for the bound, requiring softplus modifications and a tuned-free constant choice.
Experimental setup
Datasets: GuacaMol LassoBench DNA task Hartmann 6D Lunar Lander controller task Rover trajectory task
Benchmarks: Hartmann 6D Lunar Lander Rover Lasso DNA Guacamol Osimertinib MPO Guacamol Fexofenadine MPO Median Molecules 1 Median Molecules 2
Metrics: best objective value found vs number of oracle calls
Baselines: ELBO-SVGP EI Moss et al. inducing point placement exact GP EI TuRBO
Builds on: utility-calibrated variational inference one-shot knowledge gradient TuRBO SVGP q-EI/q-KG
Closest papers by meaning
- A survey and benchmark of high-dimensional Bayesian optimization of discrete sequences (NeurIPS 2024 · similarity 0.89)
- A Unified Framework for Entropy Search and Expected Improvement in Bayesian Optimization (ICML 2025 · similarity 0.88)
- Optimizing Posterior Samples for Bayesian Optimization via Rootfinding (ICLR 2025 · similarity 0.87)
- Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes (NeurIPS 2024 · similarity 0.87)
- Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs (NeurIPS 2025 · similarity 0.87)
- A General Framework for User-Guided Bayesian Optimization (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.