Research map / Reinforcement learning / Visual and representation RL
EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EARL-BO, an RL-based multi-step lookahead Bayesian optimization method using an Attention-DeepSets state encoder and model-based PPO training in a GP virtual environment, evaluated for high-dimensional synthetic and HPO-B tasks.
Problem
Multi-step lookahead Bayesian optimization is computationally intractable and approximate in high-dimensional black-box optimization due to the curse of dimensionality.
Approach
Represent the BO data state with a permutation- and size-invariant Attention-DeepSets encoder trained end-to-end with a PPO agent, using rewards from samples of the current GP posterior as a virtual environment, warm-started with TuRBO off-policy data.
Key results
- On 30-D synthetic benchmarks, EARL-BO significantly outperforms EI, TuRBO, and SAASBO across all four functions per the authors.
- On 8-D Ackley, EARL-BO (3-step) takes about 840 s per iteration versus over 3600 s for Rollout_EI (3-step), which exceeded the timeout.
- On HPO-B tasks, EARL-BO consistently outperforms baselines in regret, but performs similarly to baselines early on and with only five initial points in 19-D (ID 7200).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EARL-BO instead of EI, TuRBO, or SAASBO | Bayesian optimization · 30-D Ackley, Levy, Rosenbrock, and Sum Squares | simple regret | Helped |
| use EARL-BO instead of EI, Random Search, or TuRBO | Bayesian optimization · 5-D and 8-D Ackley, Levy, and Rosenbrock | simple regret | Helped |
| use SAASBO instead of EARL-BO | Bayesian optimization · 2-D, 5-D, and 8-D Sum Squares | simple regret | Hurt |
| use EARL-BO instead of competing BO baselines | Bayesian optimization · HPO-B search spaces 5889 (6-D), 5968 (8-D), and 7200 (19-D), with 50 initial points for 19-D | simple regret | Helped |
| use EARL-BO instead of competing BO methods | Bayesian optimization · HPO-B search space 7200 (19-D), with 5 initial samples | simple regret | No clear effect |
| increase lookahead horizon from 3 to 5 steps (ablation) | EARL-BO · 8-D synthetic benchmarks | simple regret | Mixed |
| increase lookahead horizon from 3 to 5 steps (ablation) | EARL-BO · 8-D Ackley | average runtime per BO iteration (s): 840 -> 1075 | Hurt |
| reverse the RL and encoder learning rates from (0.001, 0.01) to (0.01, 0.001) (ablation) | EARL-BO · 8-D Ackley | simple regret | Hurt |
| replace the Attention-DeepSets encoder with an attention-only encoder (ablation) | EARL-BO · 8-D Ackley, 3- and 5-step lookahead | simple regret | Hurt |
| use a 1-step lookahead horizon instead of multi-step lookahead (ablation) | EARL-BO · 8-D Ackley | simple regret | Hurt |
| increase lookahead horizon to 7 steps (ablation) | EARL-BO · 8-D Ackley | simple regret | Hurt |
| use a 3-step horizon instead of a 5-step horizon (ablation) | EARL-BO · 8-D Ackley | average runtime per BO iteration (s): 1075 -> 840 | Helped |
Surprising findings
- SAASBO outperforms EARL-BO on Sum Squares across 2-D, 5-D, and 8-D settings.
- Longer lookahead (5 or 7 steps) can degrade performance, which the authors attribute to 'planning delusion' from GP model errors.
Limitations
- Performance degrades with longer lookahead horizons due to GP model errors ('planning delusion').
- Underperforms or matches baselines when initial data is very sparse (19-D with 5 initial points).
- SAASBO is better on smooth Sum Squares function.
Experimental setup
Datasets: Ackley Levy Rosenbrock Sum Squares HPO-B
Benchmarks: Ackley Levy Rosenbrock Sum Squares HPO-B 5889 HPO-B 5968 HPO-B 7200
Metrics: simple regret
Baselines: Random search EI PI TuRBO SAASBO Rollout_VR
Builds on: PPO DeepSets TuRBO rollout-based BO with variance reduction Dyna
Closest papers by meaning
- Reinforcement Learning with Multi-Step Lookahead Information Via Adaptive Batching (NeurIPS 2026 · similarity 0.87)
- BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL (ICLR 2025 · similarity 0.86)
- MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimization (NeurIPS 2025 · similarity 0.86)
- Optimizing the Unknown: Black Box Bayesian Optimization with Energy-Based Model and Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Augmented Bayesian Policy Search (ICLR 2024 · similarity 0.85)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.