Research map / Reinforcement learning / Policy gradient and actor-critic
Augmented Bayesian Policy Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Augmented Bayesian Search (ABS), a local BO method for deterministic linear policies whose GP prior mean is an advantage-based estimate from a DroQ critic ensemble, bridging BO and deterministic policy gradient.
Problem
Direct policy search with deterministic policies in high-dimensional continuous control suffers from BO's black-box treatment of RL, which ignores MDP structure and discards trajectory information.
Approach
Use the performance difference lemma to build an advantage mean function for the GP prior on policy return, using a learned action-value function from past trajectories; the posterior gradient mean then reduces to a deterministic policy gradient plus a GP correction. Also adds an R²-weighted softmax aggregation over a critic ensemble with critic resetting.
Key results
- ABS consistently outperforms MPD across MuJoCo tasks, with relative gains increasing in high-dimensional tasks such as Walker2d and Ant.
- The validation R^2 of the advantage mean function stays positive, explaining on average about 50% of policy return variance on observed points.
- Adaptive softmax aggregation of critics outperforms average ensembling, and resetting the worst critic further improves performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the advantage mean function instead of the constant mean | MPD on MuJoCo locomotion tasks · MuJoCo-v4 locomotion tasks | policy return | Helped |
| Use ABS rather than MPD | Local Bayesian optimization · High-dimensional MuJoCo tasks, including Walker2d and Ant | policy return | Helped |
| Use ABS rather than ARS | Deterministic policy search · Walker2d and Ant | policy return | Helped |
| Use an ensemble of critics instead of a single critic (ablation) | ABS · MuJoCo locomotion tasks | policy return | Helped |
| Use adaptive aggregation instead of average ensembling (ablation) | ABS critic ensemble · MuJoCo locomotion tasks | policy return | Helped |
| Reset the least-performing critic (ablation) | ABS critic ensemble · MuJoCo locomotion tasks | policy return | Helped |
| Combine adaptive aggregation with resetting the least-performing critic (ablation) | ABS critic ensemble · MuJoCo locomotion tasks | policy return | Helped |
Surprising findings
- The test R^2 of the mean function is sometimes negative on acquisition points, yet the method still works because only one good descent direction is needed.
Limitations
- Restricted to linear policies due to computational considerations.
- Only one descent step per outer loop because the mean function depends on the central point.
- Test R^2 is sometimes negative, so the mean function does not predict returns well at all acquisition points.
Experimental setup
Datasets: MuJoCo locomotion tasks (v4)
Benchmarks: MuJoCo-v4 InvertedPendulum Swimmer Hopper HalfCheetah Walker2d Ant
Metrics: maximum discounted policy return undiscounted return validation R^2 test R^2
Baselines: MPD ARS
Builds on: Maximum Probability of Descent (MPD) Gradient Information BO (GIBO) DroQ Augmented Random Search performance difference lemma
Closest papers by meaning
- Local policy search with Bayesian optimization (NeurIPS 2021 · similarity 0.90)
- Optimizing the Unknown: Black Box Bayesian Optimization with Energy-Based Model and Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Bayesian Exploration Networks (ICML 2024 · similarity 0.88)
- Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief (ICML 2026 · similarity 0.88)
- Transition Constrained Bayesian Optimization via Markov Decision Processes (NeurIPS 2024 · similarity 0.88)
- Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.