Research map / Probabilistic and scientific modeling / Bayesian optimization
Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows standard GP-based BO with Matérn-5/2 kernels rivals or beats high-dimensional BO methods, explains SE-kernel failure via gradient vanishing from length-scale initialization, and proposes ℓ0 = c√d initialization that fixes SE-kernel BO.
Problem
Standard Bayesian optimization with GP surrogates is widely believed to fail in high-dimensional optimization, without robust empirical evidence or theoretical justification.
Approach
Attributes the SE-kernel failure in high dimensions to gradient vanishing caused by small constant length-scale initialization (e.g., 0.693 or 1), proves probabilistic tail bounds showing Matérn kernels are less susceptible, and proposes initializing length-scales as c·√d (c=1) so standard BO with SE or Matérn kernels performs competitively without extra priors.
Key results
- SBO-Matérn and SBO-SE with robust initialization achieve top-tier results across most of the twelve benchmarks, often outperforming specialized high-dimensional BO methods by a large margin on Ackley(300,150), DNA, and SVM.
- Standard SE-kernel BO with ℓ0=0.693 fails at d ≥ 150 due to gradient vanishing, while the proposed ℓ0=√d initialization removes vanishing gradients in numerical experiments.
- Theoretical threshold analysis gives dimension d=205 for SE vs d=1040 for Matérn at lower bound 0.99 when ℓ0=0.5, showing Matérn tolerates much higher dimensions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an ARD Matérn kernel instead of an ARD SE kernel with the default length-scale initialization | Standard BO · High-dimensional optimization benchmarks, including SVM (388) and Rosenbrock(300, 100) | optimization performance | Helped |
| Initialize length-scales with ℓ₀ = √d (c=1) instead of ℓ₀=0.693 (ablation) | Standard BO with the SE kernel · High-dimensional optimization benchmarks | optimization performance | Helped |
| Initialize length-scales with ℓ₀ = √d (c=1) instead of ℓ₀=0.693 (ablation) | Standard BO with the Matérn kernel · Humanoid-Standup (1003) | optimization performance | Helped |
| Use a single length-scale instead of ARD length-scales (ablation) | SBO-Matérn · Synthetic and real-world benchmarks except Humanoid-Standup (1003) | optimization performance | Mixed |
| Use UCB with λ=0.5 rather than larger tested λ values (ablation) | SBO-Matérn · SVM (388) and Ackley | optimization performance | Hurt |
| Vary UCB exploration level among λ=0.5, 1.0, 1.5, and 2.0 (ablation) | SBO-Matérn · Mopta08, Rover, NAS201, and DNA | optimization performance | No clear effect |
| Use log-EI instead of UCB (ablation) | Standard BO with the Matérn kernel · High-dimensional optimization benchmarks | optimization performance | No clear effect |
| Use EI instead of UCB (ablation) | Standard BO with the Matérn kernel · Mopta08 (128), SVM (388), DNA (180), Rosenbrock(100, 100), and Ackley | optimization performance | Hurt |
| Use Thompson sampling instead of UCB (ablation) | Standard BO with the Matérn kernel · Mopta08 (128), SVM (388), DNA (180), Rosenbrock(100, 100), and Ackley | optimization performance | Hurt |
| Use EI or Thompson sampling instead of UCB (ablation) | Standard BO with the Matérn kernel · Ackley(150, 150) and Ackley(300, 150) | optimization performance | Hurt |
| Use a BNN-based surrogate instead of a GP-based method (ablation) | BNN Laplace and BNN HMC · Synthetic and real-world benchmarks | optimization performance | Hurt |
| Use Matérn with ℓ₀=0.5 instead of SE with ℓ₀=0.5 (ablation) | GP training · Hartmann6(d, 6) and Rosenbrock(d, d) | gradient-vanishing onset dimension: 200 -> 400 | Helped |
Surprising findings
- Standard BO with Matérn kernel and a small length-scale initialization (0.693) achieves top-tier performance on nearly all benchmarks, contradicting the belief that standard BO fails above ~20 dimensions.
- Standard BO with a single shared length-scale (SBO-Single, Matérn) often closely matches ARD Matérn performance.
Limitations
- Does not analyze whether an upper limit on input dimension exists for standard BO.
- Small constant initialization can still cause vanishing gradients at extremely high dimensions (e.g., over 1K) like Humanoid-Standup.
- Optimizer choice (L-BFGS vs Adam vs RMSProp) affects GP training stability and is not fully resolved.
Experimental setup
Datasets: Ackley Rosenbrock Hartmann6 Stybtang Mopta08 SVM Rover DNA NAS201
Benchmarks: Ackley(300,150) Rosenbrock(300,100) Hartmann6(300,6) Rosenbrock(100,100) Stybtang(200,200) Ackley(150,150) Mopta08 SVM Rover DNA NAS201 Humanoid-Standup
Metrics: maximum function value obtained test MSE length-scale relative L2 difference gradient norm
Baselines: TuRBO SaasBO ALEBO HESBO RDUCB Tree UCB VBO BNN Laplace BNN HMC
Builds on: GPyTorch BoTorch TuRBO SaasBO ALEBO
Closest papers by meaning
- Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.93)
- Vanilla Bayesian Optimization Performs Great in High Dimensions (ICML 2024 · similarity 0.92)
- Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization (NeurIPS 2026 · similarity 0.90)
- Bayesian Optimization through Gaussian Cox Process Models for Spatio-temporal Data (ICLR 2024 · similarity 0.88)
- Robust and Computation-Aware Gaussian Processes (NeurIPS 2025 · similarity 0.88)
- Minimizing UCB: a Better Local Search Strategy in Local Bayesian Optimization (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.