Research map / Probabilistic and scientific modeling / Bayesian optimization
Vanilla Bayesian Optimization Performs Great in High Dimensions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that vanilla BO with a dimensionality-scaled lengthscale prior (DSP, lengthscales scaled by sqrt(D) via LogNormal prior and fixed signal variance) performs strongly on high-dimensional tasks up to 6392D, beating SAASBO, TuRBO, BAxUS and others on several benchmarks.
Problem
Bayesian optimization in high dimensions suffers from high assumed model complexity under vanilla GP priors, leading to nearly independent (uninformative) GP models and locally-restricted acquisition behaviour; existing HDBO methods impose structural assumptions to address this.
Approach
Scale the mean of the LogNormal lengthscale prior by log(D)/2 so lengthscales grow as sqrt(D), keeping the GP's assumed complexity (maximal information gain) nearly constant with dimension without imposing subspace, additive or local structure; additionally fix signal variance to 1 instead of learning it.
Key results
- On Lasso-DNA and Humanoid, Vanilla BO with the DSP substantially outperforms all baselines; it is top-two on the remaining real-world tasks except MOPTA (where SAASBO wins) and Ant (where BAxUS wins due to low-dimensional initialization).
- On sparse synthetic test functions, SAASBO performs best on average, with Vanilla BO (DSP) second, while requiring a small fraction of SAASBO's runtime.
- Empirically, global GP with fixed short lengthscales becomes approximately independent (uninformative) around 18-24 dimensions for ~1000-5000 data points, whereas the DSP keeps MIG growth flat up to D=1000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the GP lengthscale prior with dimensionality (DSP) | Vanilla BO · Lasso-DNA | best observed value | Helped |
| Scale the GP lengthscale prior with dimensionality (DSP) | Vanilla BO · Humanoid | best observed value | Helped |
| Scale the GP lengthscale prior with dimensionality (DSP) | Vanilla BO · SVM | best observed value | Helped |
| Scale the GP lengthscale prior with dimensionality (DSP) | Vanilla BO · MOPTA08 | best observed value | Mixed |
| Scale the GP lengthscale prior with dimensionality (DSP) | Vanilla BO · MuJoCo Ant | best observed value | Mixed |
| Use the DSP lengthscale prior instead of the conventional Γ(3, 6) prior | Vanilla BO · Mid-dimensional real-world tasks | best observed value | Helped |
| Increase uncertainty of the lengthscale prior while keeping its mode fixed (ablation) | DSP · SVM | best observed value | Hurt |
| Decrease uncertainty of the lengthscale prior while keeping its mode fixed (ablation) | DSP · MOPTA, Lasso-DNA, and SVM | best observed value | No clear effect |
| Assume lower complexity by increasing the default lengthscale (ablation) | DSP · Levy and Hartmann | log regret | Mixed |
| Assume higher complexity by decreasing the default lengthscale (ablation) | DSP · SVM | best observed value | Hurt |
| Learn the signal variance instead of fixing it (ablation) | DSP · Synthetic and real-world tasks | optimization performance | Mixed |
| Use conventional Γ(3, 6) lengthscale priors instead of DSP (ablation) | Vanilla BO · Hartmann | log regret | No clear effect |
Surprising findings
- Proposition shows EI does not seek high-variance boundary points when the model is uninformed; instead it queries close to the incumbent, contrary to the commonly held boundary-issue claim.
- Conventional Gamma(3,6) lengthscale MAP priors perform locally (local-search-like) yet can match DSP's final performance on some tasks such as Hartmann.
Limitations
- Assuming too low complexity can occasionally hurt performance; the approach depends on prior hyperparameters (mu_0, sigma_0) chosen for a 1D objective.
- Learning the signal variance makes performance less stable, so the outputscale is fixed to 1.
- MLE fitting is unstable in the overparameterized high-dimensional setting; SAASBO had to be run in a reduced-budget variant.
Experimental setup
Datasets: Levy (4D) Hartmann (6D) MOPTA08 Lasso-DNA SVM MuJoCo Ant MuJoCo Humanoid Lunar Lander Robot Pushing MuJoCo Swimmer MuJoCo Hopper
Benchmarks: Levy (4D) Hartmann (6D) Lunar Lander (12D) Robot Pushing (14D) Swimmer (16D) Hopper (32D) MOPTA08 (124D) Lasso-DNA (180D) SVM (388D) MuJoCo Ant (888D) MuJoCo Humanoid (6392D)
Metrics: best observed value average log regret distance to incumbent lengthscale distribution
Baselines: SAASBO TuRBO BAxUS MPD RD-UCB MCTS-VS CMA-ES Gamma(3,6) lengthscale prior MAP MLE
Builds on: Expected Improvement LogEI BoTorch SAASBO TuRBO
Closest papers by meaning
- Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.93)
- Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization (ICLR 2025 · similarity 0.92)
- BOIDS: High-Dimensional Bayesian Optimization via Incumbent-Guided Direction Lines and Subspace Embeddings (AAAI 2025 · similarity 0.90)
- Relaxing the Additivity Constraints in Decentralized No-Regret High-Dimensional Bayesian Optimization (ICLR 2024 · similarity 0.90)
- Posterior Inference with Diffusion Models for High-dimensional Black-box Optimization (ICML 2025 · similarity 0.89)
- Bayesian Optimisation with Unknown Hyperparameters: Regret Bounds Logarithmically Closer to Optimal (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.