Research map / Probabilistic and scientific modeling / Bayesian optimization
Understanding High-Dimensional Bayesian Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why simple Bayesian optimization works in high-dimensional spaces, identifies vanishing gradients from GP length-scale initialization and the role of locality (RAASP sampling), and proposes MSR, an MLE variant with scaled initial length scales plus RAASP sampling.
Problem
High-dimensional Bayesian optimization (HDBO) performance is poorly understood; recent simple BO methods perform well on real-world high-dimensional tasks, contradicting prior beliefs about the curse of dimensionality.
Approach
Vanishing gradients of the GP marginal likelihood and acquisition function under short initial length scales explain BO failures in high dimensions; scaling the initial MLE length scale by sqrt(d) and adding RAASP sampling (local search) avoids them, making plain MLE competitive with MAP-based hyperpriors.
Key results
- MSR matches state-of-the-art DSP across real-world benchmarks and outperforms DSP on 124d Mopta08 and 888d Ant.
- Constant length scale initialization (ℓ=ln 2) without RAASP shows vanishing MLE gradients and fails on 888d Ant and 6392d Humanoid.
- Benchmarks Mopta08 and Lasso-DNA have many secondary dimensions set to the boundary, suggesting lower effective dimensionality than nominal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MSR instead of DSP | BO with GP surrogate · Mopta08 (d=124) | final performance rank: 2 -> 1 | Helped |
| use MSR instead of DSP | BO with GP surrogate · Lasso-DNA (d=180) | final performance rank: 1 -> 4 | Hurt |
| use MSR instead of DSP | BO with GP surrogate · Ant (d=888) | final performance rank: 4 -> 2 | Helped |
| use MSR instead of DSP | BO with GP surrogate · Humanoid (d=6392) | final performance rank: 1 -> 2 | Hurt |
| randomize dominant dimensions (ablation) | DSP solutions on Lasso-DNA · Lasso-DNA (d=180) | average objective value: 0.315 -> 0.315 (± \num{4e-4}) | Hurt |
| randomize dominant dimensions (ablation) | DSP solutions on Mopta08 · Mopta08 (d=124) | average objective value: 328.824 -> 328.824 ($\pm 1.920 ) | Hurt |
| randomize secondary dimensions (ablation) | DSP solutions on Lasso-DNA · Lasso-DNA (d=180) | average objective value: 0.315 -> 0.445 | Hurt |
| randomize secondary dimensions (ablation) | DSP solutions on Mopta08 · Mopta08 (d=124) | average objective value: 328.824 -> 430.474 | Hurt |
| add RAASP sampling to scaled MLE (ablation) | MLE (scaled) · Ant and Humanoid | length-scale estimate consistency | Helped |
Surprising findings
- Plain MLE with scaled initial length scales achieves state-of-the-art HDBO performance without a MAP hyperprior.
- Popular HDBO benchmarks Mopta08 and Lasso-DNA have simple structure: many dimensions lie at the boundary, so they are effectively lower-dimensional.
Limitations
- Focuses only on GP surrogate models; findings not verified for random forests or Bayesian neural networks.
- Results on a limited set of benchmarks (four real-world plus synthetic functions); some benchmarks found to be overly simple.
Experimental setup
Datasets: Mopta08 Lasso-DNA SVM Rover Mujoco Ant Mujoco Humanoid Levy Schwefel Griewank
Benchmarks: Mopta08 (124d) Lasso-DNA (180d) Ant (888d) Humanoid (6392d) Levy100 Schwefel100 Griewank100
Metrics: best objective value found vs iterations final ranking average distance traveled by AF optimizer gradient magnitude OTSD exploration
Baselines: DSP TuRBO SAASBO Bounce MLE (ℓ=ln 2) CMA-ES
Builds on: TuRBO BoTorch LogEI SAASBO Dimensionality-scaled prior (DSP)
Closest papers by meaning
- Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization (ICLR 2025 · similarity 0.93)
- Vanilla Bayesian Optimization Performs Great in High Dimensions (ICML 2024 · similarity 0.93)
- Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes (NeurIPS 2024 · similarity 0.90)
- Minimizing UCB: a Better Local Search Strategy in Local Bayesian Optimization (NeurIPS 2024 · similarity 0.90)
- Informed Initialization for Bayesian Optimization and Active Learning (NeurIPS 2025 · similarity 0.89)
- BOIDS: High-Dimensional Bayesian Optimization via Incumbent-Guided Direction Lines and Subspace Embeddings (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.