Research map / Probabilistic and scientific modeling / Bayesian optimization
Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAKE, which uses an LLM (gpt-4o-mini) as crossover and mutation operators to evolve Gaussian process kernels during Bayesian optimization, plus BAKER to rank kernels by BIC-weighted expected improvement.
Problem
Bayesian optimization performance depends heavily on the GP kernel, but fixed or heuristic kernel choices often converge slowly or suboptimally when poorly matched to the objective, and data are too scarce for traditional kernel-search methods.
Approach
Use an LLM, prompted with observed data and kernel fitness, as genetic crossover and mutation operators over a kernel grammar (base kernels SE, PER, LIN, RQ, M3, M5 combined by + and ×), and select the kernel at each iteration via BAKER, which weights kernels by BIC and their expected improvement at the proposed point.
Key results
- On HPOBench across 60 tasks, CAKE + BAKER achieves the best average rank of 1.04, versus 2.40 for the next best (CAKE + Utility) and 6.80 for random sampling.
- In photonic chip design, CAKE found a solution with significantly higher score in under 40 trials, described as roughly a tenfold speedup over baselines.
- Ablation shows the LLM-based crossover/mutation outperforms random recombination and genetic algorithm in raising population fitness, and removing the instruction prompt yields only 68% valid kernels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine BIC and acquisition value for kernel ranking instead of using acquisition value only (ablation) | CAKE on HPOBench · HPOBench | average rank: 2.40 -> 1.04 | Helped |
| combine BIC and acquisition value for kernel ranking instead of using BIC only (ablation) | CAKE on HPOBench · HPOBench | average rank: 3.02 -> 1.04 | Helped |
| use LLM kernel edits instead of genetic algorithm operators (ablation) | CAKE on HPOBench · HPOBench | average rank: 2.70 -> 1.04 | Helped |
| use LLM-guided kernel generation instead of random kernel combinations (ablation) | CAKE on HPOBench · HPOBench | average rank: 6.80 -> 1.04 | Helped |
| remove optimization history from the prompt (ablation) | CAKE on HPOBench · HPOBench | optimization performance | Hurt |
| remove kernel-generation instructions from the prompt (ablation) | CAKE on HPOBench · HPOBench | optimization performance: 68% | Hurt |
| remove the reasoning requirement from the prompt (ablation) | CAKE on HPOBench · HPOBench | optimization performance | Hurt |
Surprising findings
- Without kernel-generation instructions, the LLM produced valid kernels only 68% of the time.
Limitations
- Increasing crossovers raises LLM API calls and cost
- Evaluated on specific BO benchmarks; the authors aim to extend to other ML tasks
Experimental setup
Datasets: HPOBench tabular hyperparameter optimization tasks (12 OpenML datasets, 5 ML models) robot pushing benchmark lunar lander (Gymnasium) photonic chip physics simulation
Benchmarks: HPOBench robot pushing lunar lander photonic chip design synthetic test functions (Ackley, Branin, etc.)
Metrics: average test accuracy average rank average reward score and hypervolume normalized regret
Baselines: SE kernel Matern-5/2 kernel Adaptive kernel selection (Random, Utility, BIC) Deep GP Ensemble GP Compositional Kernel Search Automated BO Single-Task GP Additive GP
Builds on: kernel grammar automated bayesian optimization genetic algorithm BoTorch
Closest papers by meaning
- Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization (NeurIPS 2026 · similarity 0.92)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.89)
- Kernel-based guarantees for nonlinear parametric models in Bayesian optimization (NeurIPS 2026 · similarity 0.88)
- CoffeeBoost: Gradient Boosting Native Conformal Inference for Bayesian Optimization (AAAI 2025 · similarity 0.87)
- BARK: A Fully Bayesian Tree Kernel for Black-box Optimization (ICML 2025 · similarity 0.87)
- Omnipresent Yet Overlooked: Heat Kernels in Combinatorial Bayesian Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.