Research map / Large language models / Agents, code and math
Large Language Models to Enhance Bayesian Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLAMBO, which frames Bayesian optimization components (warmstarting, surrogate modeling, candidate sampling) as in-context natural-language prompts to GPT-3.5, improving sample efficiency for hyperparameter tuning.
Problem
Bayesian optimization of expensive black-box functions is sample-inefficient in the early, few-observation stage, where surrogate models are inaccurate and priors are hard to encode; the paper asks whether LLMs can enhance BO components and work end-to-end.
Approach
Translate BO components into natural language so a frozen LLM performs zero-shot warmstarting, few-shot surrogate prediction (discriminative regression with MC sampling plus example shuffling, or generative good/bad classification), and target-value-conditioned candidate sampling, all via in-context learning without finetuning.
Key results
- Zero-shot LLM warmstarting with partial or full data-card context improves early search over random, Sobol and Latin Hypercube initialization for GP and TPE on 74 tasks, especially for trials under 5.
- The discriminative LLAMBO surrogate attains lower NRMSE and higher R^2 than GP and SMAC at small sample sizes, but GP and SMAC give better uncertainty calibration (LPD and coverage).
- End-to-end LLAMBO achieves the best average regret across 50 HPT tasks (25 Bayesmark, 3 proprietary, 2 synthetic in the main setup), with the largest gains in early trials.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use zero-shot LLM warmstarting instead of random initialization | GP and TPE · Bayesmark warmstarting tasks | average regret | Helped |
| add dataset meta-features to zero-shot warmstarting prompts | GP and TPE · Bayesmark warmstarting tasks | average regret | Helped |
| shuffle few-shot examples when estimating the discriminative surrogate (ablation) | LLAMBO discriminative surrogate · Discriminative surrogate evaluation | uncertainty calibration | Helped |
| use Monte Carlo sampling without shuffling few-shot examples (ablation) | LLAMBO discriminative surrogate · Discriminative surrogate evaluation | uncertainty calibration | Hurt |
| include the problem description and actual hyperparameter names (ablation) | LLAMBO discriminative surrogate · Discriminative surrogate evaluation | prediction performance | Helped |
| condition candidate sampling on desired objective values using ICL | LLAMBO candidate sampler · Candidate sampling | average regret | Helped |
| increase α from -0.5 toward 0 (ablation) | LLAMBO candidate sampler · Candidate sampling | average regret | Helped |
| increase α beyond 0 (ablation) | LLAMBO candidate sampler · Candidate sampling | average regret | Hurt |
| set α to 0.01 to extrapolate beyond observed distributions (ablation) | LLAMBO candidate sampler · Candidate sampling | best regret | Helped |
| include problem description and hyperparameter names in candidate-sampling prompts (ablation) | LLAMBO candidate sampler · Candidate sampling | average regret | Helped |
| include non-formatting task instructions for candidate generation (ablation) | LLAMBO end-to-end · Bayesmark RandomForest tasks | average regret | Helped |
| include non-formatting task instructions for candidate generation (ablation) | LLAMBO candidate sampler · Bayesmark RandomForest tasks | acceptance rate: 69.26%±0.79% -> 91.60%±0.45% | Helped |
Surprising findings
- Even with no dataset context, LLM warmstarting outperforms random initialization, suggesting generic hyperparameter correlation knowledge.
- Monte Carlo sampling alone gives poorly calibrated uncertainty; shuffling few-shot example order improves calibration and prediction.
Limitations
- Inference through GPT-3.5 is far more computationally expensive than classical BO.
- Results depend on the choice of LLM; only GPT-3.5 was evaluated.
- Evaluation is limited to relatively low-dimensional hyperparameter tuning tasks.
Experimental setup
Datasets: Bayesmark HPOBench SEER MAGGIC CUTRACT Rosenbrock Griewank KTablet
Benchmarks: Bayesmark HPOBench SEER MAGGIC CUTRACT synthetic Rosenbrock/Griewank/KTablet tasks
Metrics: normalized average regret best regret NRMSE R^2 log predictive density empirical coverage sharpness generalized variance log-likelihood acceptance rate
Baselines: GP SMAC TPE Random Sobol Latin Hypercube GP-DKL SKOpt Optuna SMAC3 TuRBO HEBO
Builds on: Tree-structured Parzen Estimator SMAC Gaussian processes Bayesmark HPOBench
Closest papers by meaning
- Model Fusion through Bayesian Optimization in Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.90)
- FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch (ICML 2025 · similarity 0.89)
- Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH (NeurIPS 2026 · similarity 0.89)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers (ICML 2024 · similarity 0.89)
- BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.