Research map / Probabilistic and scientific modeling / Bayesian optimization
Kernel Learning for Sample Constrained Black-Box Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KOBO, which learns a Gaussian process kernel by encoding grammar-composed kernels in a VAE latent space and optimizing model evidence there, reducing function-evaluation budgets for black-box optimization.
Problem
Black-box optimization with a tight sample budget, where the GP kernel must be chosen without domain knowledge; discrete kernel search is hard to optimize and sample-expensive.
Approach
Synthesize composite kernels via a context-free grammar plus a data-based distance representation, train a VAE to create a continuous latent kernel space, and run an auxiliary GPR over that space (maximizing model evidence) to pick the kernel used by the function GPR.
Key results
- On the Staircase function KOBO attains the global minimum in about 17 function evaluations versus 28 for MCMC, 32 for BOMS and 43 for CKS.
- In the hearing-loss audio personalization study, KOBO reaches regret/rating 10 for all three shown users at Q=20-25 where SE and PER lag (e.g., U3 SE 9 at Q=25 vs KOBO 10 at Q=20).
- In the random-corruption audio study, KOBO reaches 10 for U2 at Q=20 while SE and PER stay at 4 and 9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KOBO rather than MCMC | black-box optimization · Staircase | function evaluations to global minimum: 28 -> 17 | Helped |
| Use KOBO rather than BOMS | black-box optimization · Staircase | function evaluations to global minimum: 32 -> 17 | Helped |
| Use KOBO rather than CKS | black-box optimization · Staircase | function evaluations to global minimum: 43 -> 17 | Helped |
| Use KOBO instead of static SE and PER kernels | audio personalization · Hearing Loss, U1, Q=25 | user rating: 10; 10; 10 | No clear effect |
| Use KOBO instead of static SE or PER kernels | audio personalization · Hearing Loss, U2, Q=25 | user rating: 10; 10; 10 | No clear effect |
| Use KOBO instead of static SE | audio personalization · Hearing Loss, U3, Q=25 | user rating: 9 -> 10 | Helped |
| Use KOBO instead of static PER | audio personalization · Random Audio Corruption, U1, Q=25 | user rating: 9 -> 10 | Helped |
| Use KOBO instead of static SE | audio personalization · Random Audio Corruption, U1, Q=25 | user rating: 8 -> 10 | Helped |
| Use KOBO instead of static SE or PER kernels | audio personalization · Random Audio Corruption, U2, Q=25 | user rating: 8 -> 10; 9 -> 10 | Helped |
| Use KOBO instead of static SE and PER kernels | audio personalization · Random Audio Corruption, U3, Q=25 | user rating: 10 -> 8; 7 -> 8 | Mixed |
| Use grammar-based encoding length L=5 rather than L=15 (ablation) | KOBO · Staircase | regret | Hurt |
| Use grammar-based encoding length L=25 rather than L=15 (ablation) | KOBO · Staircase | regret | No clear effect |
Surprising findings
- KOBO's kernel learning gains are small on smooth functions such as Branin, where SE and PER already fit the shape.
- Learned kernels can overfit to more complex kernels than the true target function, needing more samples to converge.
Limitations
- Extra computation for model-evidence evaluation in the latent kernel search.
- Learned latent kernel space is abstract and not interpretable.
- Kernel complexity can overfit simple target functions, needing regularization.
Experimental setup
Datasets: synthetic GP-sampled functions Staircase Branin Michalewicz CO2 emissions data NHANES hearing loss profiles
Benchmarks: Staircase Branin Michalewicz Styblinski-Tang Hartmann-6 CO2 emissions audio personalization user study (6 volunteers) prompt-based image generation user study
Metrics: regret model evidence
Baselines: MCMC kernel search BOMS CKS SE kernel PER kernel
Builds on: Automatic Statistician BOMS ALEBO Bayesian optimization
Closest papers by meaning
- PG-LBO: Enhancing High-Dimensional Bayesian Optimization with Pseudo-Label and Gaussian Process Guidance (AAAI 2024 · similarity 0.85)
- Posterior Inference in Latent Space for Scalable Constrained Black-box Optimization (NeurIPS 2026 · similarity 0.85)
- Posterior Inference with Diffusion Models for High-dimensional Black-box Optimization (ICML 2025 · similarity 0.85)
- Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization (NeurIPS 2026 · similarity 0.85)
- Scalable Kernel Inverse Optimization (NeurIPS 2024 · similarity 0.84)
- Sharpness-Aware Black-Box Optimization (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.