Research map / Probabilistic and scientific modeling / Bayesian optimization
Towards AutoAI: Optimizing a Machine Learning System with Black-box and Differentiable Components
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A-BAD-BO, which optimizes a mixed ML/black-box system by running Bayesian optimization over local loss configurations and recovering system parameters via gradient descent on each ML component, with a no-regret guarantee.
Problem
Jointly optimizing complex systems that contain multiple differentiable ML components and black-box (non-differentiable, non-analytical) components, which is high-dimensional and has no analytical form.
Approach
Reparameterize system optimization as a bilevel problem over local loss configurations: BO searches the low-dimensional local loss space at the outer level, while gradient descent on each ML component's local loss finds system parameters matching a chosen configuration at the inner level, with the min over k samples as the noisy objective estimate.
Key results
- A-BAD-BO attains lower system loss than Local GD across all four systems within 200 system queries.
- A-BAD-BO outperforms GP-LCB and TuRBO after the same number of system queries, indicating better sample efficiency.
- Theory shows sub-linear attained cumulative regret (~R_T/T -> 0) when the objective has bounded RKHS norm, implying asymptotic convergence to optimal system parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use A-BAD-BO instead of independent local gradient descent | Synthetic, MNIST, Healthcare, and LLM systems · Four experimental systems | system loss | Helped |
| Use A-BAD-BO instead of GP-LCB or TuRBO | Synthetic, MNIST, Healthcare, and LLM systems · Four experimental systems | system loss | Helped |
| Increase sampling size k while keeping the number of BO iterations fixed (ablation) | A-BAD-BO · Synthetic, MNIST, Healthcare, and LLM systems | system loss | Helped |
| Use a small sampling size, such as k = 1, instead of larger sampling sizes (ablation) | A-BAD-BO · MNIST, Healthcare, and LLM systems | system loss | Helped |
| Increase sampling size k and the number of BO iterations (ablation) | A-BAD-BO · Synthetic, MNIST, Healthcare, and LLM systems | system loss | Helped |
| Choose moderate sampling size k and BO iteration count T rather than extreme values at a fixed query budget (ablation) | A-BAD-BO · Synthetic, MNIST, Healthcare, and LLM systems | system loss | Helped |
Surprising findings
- Small sampling size (k=1) still outperforms local and global baselines on MNIST, Healthcare and LLM systems, suggesting the local loss configuration is a strong information source.
Limitations
- Problem dimensionality scales with the number of ML components in the system.
- Theoretical guarantees assume noiseless black-box components; noisy extension lacks closed-form error bounds.
- For large neural networks, BO is restricted to the output layer parameters.
Experimental setup
Datasets: MNIST Hepatitis dataset (UCI) synthetic system data noisy text examples for LLM prompt engineering
Benchmarks: Synthetic system MNIST system Healthcare system LLM prompt engineering system
Metrics: system loss cumulative regret convergence vs. number of system queries
Baselines: Local GD GP-LCB TuRBO
Builds on: Bayesian optimization GP-LCB (Srinivas et al., 2010) TuRBO IGP-LCB (Chowdhury & Gopalan, 2017)
Closest papers by meaning
- Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks (NeurIPS 2021 · similarity 0.86)
- BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL (ICLR 2025 · similarity 0.85)
- Vanilla Bayesian Optimization Performs Great in High Dimensions (ICML 2024 · similarity 0.84)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.83)
- PG-LBO: Enhancing High-Dimensional Bayesian Optimization with Pseudo-Label and Gaussian Process Guidance (AAAI 2024 · similarity 0.83)
- Leaving the Nest: Going beyond Local Loss Functions for Predict-Then-Optimize (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.