Research map / Reinforcement learning / Policy gradient and actor-critic
Bayesian Exploration Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bayesian Exploration Networks (BEN), a model-free Bayesian RL method using normalising flows for aleatoric (density estimation) and epistemic (variational inference) uncertainty over the Bellman operator, with theory showing existing model-free BRL methods are not Bayes-optimal.
Problem
Model-free Bayesian reinforcement learning lacks theoretical grounding; existing model-free approaches (BootDQN+Prior, BBAC, myopic and QBRL methods) may not learn Bayes-optimal policies, and exact Bayes-optimal planning is intractable.
Approach
Characterise uncertainty over the optimal Bayesian Bellman operator rather than transition dynamics, using one normalising flow for aleatoric uncertainty (density estimation over a 1-D Bellman variable) and another flow for epistemic uncertainty (variational posterior), trained jointly with partial SGD updates so that the Bayesian Bellman equation is satisfied in the limit of full optimisation.
Key results
- In the episodic no-prior setting, BEN solves the search and rescue task while BootDQN+Prior cannot learn to listen and fails to solve it.
- With a weak prior, BEN explores in early episodes and then finds all victims immediately in later episodes.
- In the zero-shot strong-prior setting, BEN rescues all victims and avoids all hazards in a single episode, while QBRL-style oracle policies repeatedly hit hazards.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BEN instead of RNN PPO | Search and rescue, episodic tabula rasa · Search and rescue | cumulative return | Helped |
| Use BEN instead of BootDQN+prior | Search and rescue, episodic tabula rasa · Search and rescue | cumulative return | Helped |
| Use BEN with a weak prior | Search and rescue, episodic setting · Search and rescue | return | Helped |
| Use BEN instead of the QBRL variant | Search and rescue, zero-shot setting with a strong prior · Search and rescue | cumulative return | Helped |
| Increase aleatoric network capacity (ablation) | BEN · Search and rescue, zero-shot setting | return | Mixed |
| Add prior pre-training MSBBE minimisation steps (ablation) | BEN · Search and rescue, zero-shot setting | return | Helped |
Surprising findings
- Existing QBRL methods (BootDQN+Prior, BBAC) can be arbitrarily Bayes-suboptimal; in the tiger problem their optimal policies have empty intersection with Bayes-optimal policies.
- BootDQN+Prior outperforms RNN PPO in the episodic setting but still fails to learn to listen.
Limitations
- Exact Bayes-optimality holds only in the limit of complete optimisation; partial optimisation yields only approximately Bayes-optimal policies.
- Experiments are limited to a small custom gridworld (7x7) and tiger-style problems.
- Performance depends on capacity of the aleatoric flow, requiring a balance between expressiveness and generalisation.
Experimental setup
Datasets: search and rescue gridworld (7x7, 8 hazards, 4 victims)
Benchmarks: search and rescue gridworld
Metrics: cumulative return return per episode
Baselines: RNN PPO BootDQN+Prior QBRL variant of BEN
Builds on: BootDQN+Prior VariBAD Bayes-adaptive MDP normalising flows tiger problem
Closest papers by meaning
- Bayesian Policy Optimization for Model Uncertainty (ICLR 2019 · similarity 0.91)
- Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis (AAAI 2024 · similarity 0.89)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.89)
- Augmented Bayesian Policy Search (ICLR 2024 · similarity 0.88)
- Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens (ICML 2025 · similarity 0.88)
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.