Research map / Probabilistic and scientific modeling / Bayesian optimization
Joint Composite Latent Space Bayesian Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JoCo, which jointly trains neural encoders for high-dimensional inputs and intermediate outputs together with GP surrogates in a composite BO loop, outperforming high-dimensional BO baselines on nine composite tasks including LLM and text-to-image adversarial prompt attacks.
Problem
Bayesian optimization of composite functions f = g∘h where both the input x and the intermediate output h(x) are high-dimensional, making standard GP surrogates and prior composite BO methods ineffective or intractable.
Approach
Jointly train input and outcome neural encoders with multi-output GP surrogates (for h in latent space) and a GP on the latent outcome (for f), using the composite objective as supervision, and update all components continuously during BO with Thompson sampling inside a TuRBO trust region.
Key results
- JoCo outperforms baselines (Vanilla BO w/ DKL, TuRBO, CMA-ES, random) across all nine high-dimensional composite BO tasks, with especially strong early-optimization gains on toxic text and image generation.
- On held-out RMSE, the JoCo architecture beats GP+DKL and plain GP on all nine tasks (e.g., Dog Image Gen 1.844 vs 5.741; Env Model 4.191 vs 6.351).
- Removing joint training, continuous model updating, or trust regions significantly degrades JoCo performance, especially on generative AI tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JoCo's jointly trained latent encoders and probabilistic models instead of a standard approximate GP | JoCo architecture · All nine held-out task datasets | RMSE | Helped |
| Add a deep kernel to the approximate GP | Approximate GP · All nine held-out task datasets | RMSE | Helped |
| Remove continuous model updates during optimization (ablation) | JoCo · Optimization tasks, especially generative AI tasks | optimization performance | Hurt |
| Replace joint updates with separate updates of the input/outcome model and reward model (ablation) | JoCo · Optimization tasks | optimization performance | Hurt |
| Remove the TuRBO trust region (ablation) | JoCo · All tasks | optimization performance | Hurt |
| Do not propagate uncertainty through estimated outcomes (ablation) | JoCo · Optimization tasks | optimization performance | Hurt |
| Do not propagate uncertainty through estimated rewards (ablation) | JoCo · Optimization tasks | optimization performance | Hurt |
| Use expected improvement instead of Thompson sampling (ablation) | JoCo · Optimization tasks | optimization performance | Hurt |
| Vary the outcome encoder's latent dimension (ablation) | JoCo · Toxic Text Gen | optimization performance | No clear effect |
| Vary the number of update epochs, using a nonzero number of epochs (ablation) | JoCo · Optimization tasks | optimization performance | No clear effect |
| Vary the number of recent data points used for updates (ablation) | JoCo · Optimization tasks | optimization performance | No clear effect |
| Vary the learning rate (ablation) | JoCo · Optimization tasks | optimization performance | No clear effect |
Surprising findings
- On some synthetic and PDE tasks, baselines such as TuRBO eventually catch up to JoCo after many evaluations, with JoCo's advantage concentrated early in optimization.
Limitations
- Compositional BO baseline (Maus et al./EI-CF) is intractable for high-dimensional outputs, so not compared directly on most tasks.
- Some baselines catch up to JoCo after sufficiently many evaluations on several tasks.
- Requires choosing encoder architectures and hyperparameters per task (e.g., latent dimension 32), with limited theoretical analysis.
Experimental setup
Datasets: synthetic composite Langermann and Rosenbrock functions environmental modeling problem Brusselator PDE task rover trajectory planning task toxic text classifier and Emotion English sentiment classifier ImageNet ResNet18 classifier objective
Benchmarks: Langermann Rosenbrock Env Model PDE Rover Toxic Text Gen Aircraft Image Gen Dog Image Gen Sportscar Image Gen
Metrics: best objective value vs function evaluations RMSE of held-out predictions runtime and memory
Baselines: Vanilla BO with DKL TuRBO CMA-ES random sampling EI-CF TS-CF SAASBO
Builds on: TuRBO deep kernel learning compositional Bayesian optimization parametric GP regression
Closest papers by meaning
- BOIDS: High-Dimensional Bayesian Optimization via Incumbent-Guided Direction Lines and Subspace Embeddings (AAAI 2025 · similarity 0.88)
- Posterior Inference with Diffusion Models for High-dimensional Black-box Optimization (ICML 2025 · similarity 0.88)
- Latent Bayesian Optimization via Autoregressive Normalizing Flows (ICLR 2025 · similarity 0.87)
- Informed Initialization for Bayesian Optimization and Active Learning (NeurIPS 2025 · similarity 0.87)
- Bayesian Optimization on Function Spaces via Sparse RKHS Manifolds (NeurIPS 2026 · similarity 0.87)
- Batched Energy-Entropy acquisition for Bayesian Optimization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.