Research map / Probabilistic and scientific modeling / Bayesian optimization
Modeling All Response Surfaces in One for Conditional Search Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AttnBO, a deep-kernel Gaussian process with a structure-aware hyperparameter embedding and attention encoder that models all subspaces of a conditional (tree-structured) search space with one shared surrogate.
Problem
Bayesian optimization over conditional search spaces, where configurations have different structures and dimensionalities across subspaces, so standard GPs do not apply and separate per-subspace GPs are sample-inefficient.
Approach
Embed each hyperparameter as a token carrying identity, index, value, and parent-identity embeddings; encode configurations of varying length with a Transformer encoder and average pooling into a unified latent space, then place a single standard Matérn GP on top (deep kernel learning), allowing relationships across subspaces to be shared.
Key results
- On the simulation function, AttnBO reaches within 100 observations the performance separate GP models need 200 observations to reach.
- On three OpenML conditional spaces (SVM, XGBoost, CASH), AttnBO achieves the best average ranking across all three tasks, outperforming AddTree.
- On the NAS space, AttnBO supports batch queries (one per subspace) and outperforms AddTree, which only gives one query per iteration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use structure-aware hyperparameter embeddings instead of values alone (ablation) | AttnBO · SVM and XGBoost tasks, average ranking over 5 datasets | ranking | Helped |
| replace average pooling with a global-feature token (AttnBO-token-mixer) (ablation) | AttnBO · SVM and XGBoost tasks, average ranking over 5 datasets | ranking | Mixed |
| vary attention-block count, head count, and MLP configuration (ablation) | AttnBO · SVM and XGBoost tasks | accuracy | No clear effect |
| use a unified surrogate instead of separate Gaussian Processes | AttnBO · conditional simulation objective function | observations to reach the same performance: 100 observation points vs 200 points | Helped |
| add warm-starting from HPO-B training data (ablation) | AttnBO · HPO-B-v3 | performance | Helped |
| use AttnBO rather than Bandits-BO or Add-tree | conditional simulation task · simulation function | performance | Helped |
| use AttnBO rather than existing methods | NAS task · CIFAR-10 NAS | optimal accuracy | Mixed |
| use AttnBO rather than other methods | SVM, XGBoost, and CASH search spaces · OpenML datasets | average ranking | Helped |
Surprising findings
- Random Search performed better than several BO methods, including AddTree, on the NAS task due to exploration across a larger space with parallelism.
Limitations
- Computational overhead from training the deep encoder, slightly higher than other baselines.
- Acquisition optimization per subspace with lbfgs becomes expensive when the number of subspaces is large; random search or Thompson sampling is needed instead.
Experimental setup
Datasets: simulation tree-structured function OpenML tasks (10101, 37, 9967, 9946, 10093, 3494) CIFAR-10 (NAS) HPO-B-v3
Benchmarks: simulation tree-structured function NAS on CIFAR-10 OpenML SVM/XGBoost/CASH tasks HPO-B-v3
Metrics: log10 regret average ranking accuracy time cost
Baselines: Random Search Bandits-BO AddTree SMAC TPE AttnBO-no-emb AttnBO-token-mixer
Builds on: deep kernel learning Add-Tree Bandits-BO transformer
Closest papers by meaning
- Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning (NeurIPS 2019 · similarity 0.90)
- Understanding High-Dimensional Bayesian Optimization (ICML 2025 · similarity 0.87)
- Informed Initialization for Bayesian Optimization and Active Learning (NeurIPS 2025 · similarity 0.86)
- BOIDS: High-Dimensional Bayesian Optimization via Incumbent-Guided Direction Lines and Subspace Embeddings (AAAI 2025 · similarity 0.85)
- High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling (ICML 2024 · similarity 0.85)
- Joint Composite Latent Space Bayesian Optimization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.