Research map / Probabilistic and scientific modeling / Bayesian optimization

Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization

NeurIPS 2026 · Probabilistic and scientific modeling › Bayesian optimization · approach: Local and high-dimensional Bayesian optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Kernel Discovery, an LLM-driven evolutionary framework that searches beyond additive/multiplicative kernel compositions for high-dimensional BO, using a two-stage (math form then code) LLM generation and a LOO-CRPS selection criterion.

Problem

Designing effective GP kernels for high-dimensional Bayesian optimization relies on manual engineering; existing automated kernel search is limited to additive/multiplicative compositions, and LLM-based BO methods condition on raw observations that do not scale to high dimensions.

Approach

An LLM first proposes novel mathematical kernel forms (without conditioning on raw observations), a second LLM call converts each form to validated GPyTorch code, and kernels are selected per iteration by leave-one-out CRPS instead of marginal likelihood.

Key results

• Kernel Discovery achieves an average rank of 1.2 out of 17 methods across five high-dimensional BO benchmarks.
• Best method on Rover (4.353), Lasso-DNA (0.286), SVM (0.056) and Humanoid (762.78); runner-up Compositional Search has average rank 4.4.
• CAKE, which conditions the LLM on raw observations, ranks 9.8 and underperforms several base kernels.

Surprising findings

• On the Ant benchmark, a zero-initialization policy outperforms all GP-BO kernel methods, so Ant was excluded.
• Directly prompting an LLM for kernel code yields functionally redundant kernels, motivating the two-stage math-then-code approach.

Limitations

• Discovery runs independently per benchmark, so kernels are not optimized across tasks; transfer is shown only for one kernel.
• A non-trivial fraction of LLM-generated kernels fail PSD or dimension-agnosticity checks.
• Ant benchmark was excluded because zero initialization dominates, limiting the evaluation set to five tasks.

Experimental setup

Datasets: Rover Mopta08 Lasso-DNA SVM Humanoid

Benchmarks: Rover Mopta08 Lasso-DNA SVM Humanoid

Metrics: best value found average rank across methods

Baselines: RBF Matérn52 Linear Periodic RQ BOCK SL Greedy Search Compositional Search CAKE ARM ATRBO

Builds on: CAKE LLaMEA-BO BOCK RAASP

Closest papers by meaning

• Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs (NeurIPS 2025 · similarity 0.92)
• Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization (ICLR 2025 · similarity 0.90)
• Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.89)
• BARK: A Fully Bayesian Tree Kernel for Black-box Optimization (ICML 2025 · similarity 0.88)
• Kernel-based guarantees for nonlinear parametric models in Bayesian optimization (NeurIPS 2026 · similarity 0.88)
• Vanilla Bayesian Optimization Performs Great in High Dimensions (ICML 2024 · similarity 0.87)