Research map / Probabilistic and scientific modeling / Bayesian optimization
Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an information-geometric view of BO (pullback Fisher tensor through the surrogate posterior map) that bounds acquisition gradients, and builds FITR, a TuRBO-style trust region whose axis-aligned shape uses diagonal regularized pullback-Fisher weights instead of GP lengthscales.
Problem
Acquisition-function optimization in Bayesian optimization becomes hard in high dimensions due to vanishing gradients, and existing fixes (dimension-scaled lengthscale priors, RAASP, trust regions) lack a common explanation and are tied to lengthscale-based GP kernels.
Approach
Pull back the Fisher-Rao metric of the predictive distribution manifold through the surrogate's posterior map to get a local sensitivity tensor on input space; this bounds the acquisition gradient norm and gives a trust-region shape from local Fisher weights, applicable to any differentiable probabilistic surrogate.
Key results
- On GP-SE benchmarks, FITR-REI improves on TuRBO-REI on several problems and remains competitive with DSP and BOUNCE on the rest (mean over 11 seeds).
- Full-tensor FullFITR did not outperform diagonal FITR on the subset where it was evaluated.
- With the IBNN kernel, FITR improves on the non-trust-region baseline on a subset of tasks and is comparable on the remainder (5 seeds).
Surprising findings
- The full whitening-based FITR variant did not outperform the diagonal approximation.
- A KL-constrained trust region solved with an augmented Lagrangian method became overly exploitative and degraded performance.
Limitations
- Proposition bounds acquisition gradient during inner optimization but gives no regret or global convergence guarantee.
- For scalar predictive families the pullback rank is at most two, so the full tensor is rank-deficient; regularization needed.
- Gains for non-isotropic IBNN kernels are task-dependent and the vanishing-gradient argument applies less sharply.
Experimental setup
Datasets: HPA102-1 MOPTA08 LassoDNA SVM control and robotics benchmarks (nine continuous problems)
Benchmarks: HPA102-1 MOPTA08 LassoDNA SVM nine continuous GP-SE benchmark problems
Metrics: best observed objective value trust-region length candidate generation time anisotropy ratio
Baselines: TuRBO-REI DSP BOUNCE IBNN non-trust-region baseline FullFITR
Builds on: TuRBO REI DSP LogEI BoTorch
Closest papers by meaning
- High-Dimensional Bayesian Optimization via Nested Riemannian Manifolds (NeurIPS 2020 · similarity 0.89)
- Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.88)
- Vanilla Bayesian Optimization Performs Great in High Dimensions (ICML 2024 · similarity 0.88)
- A Unified Framework for Entropy Search and Expected Improvement in Bayesian Optimization (ICML 2025 · similarity 0.88)
- A General Framework for User-Guided Bayesian Optimization (ICLR 2024 · similarity 0.87)
- Bayesian Optimisation with Unknown Hyperparameters: Regret Bounds Logarithmically Closer to Optimal (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.