Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Epistemic Bellman Operators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Epistemic Bellman Operators (EBOs), operators on distributions over Q-values that unify Bayesian/ensemble deep RL target handling, proves they are contractions, and builds an uncertainty-aware PPO variant (ECPPO) that adapts its clipping range.
Problem
Bayesian and ensemble model-free RL algorithms condition posteriors on distributions over target parameters, but there is no guarantee that repeatedly alternating posterior inference and target updates converges, and methods treat target uncertainty inconsistently.
Approach
Define a Bellman operator on distributions: pushforward of a contractive Bellman operator with additive noise, marginalized over a distribution of target Q-values. Show it is a Wasserstein contraction, so interleaved posterior inference and target updates converge to a unique fixed point; apply it to Thompson sampling analysis and to PPO's advantage uncertainty for adaptive clipping.
Key results
- Epistemic Bellman operators are proven Wasserstein contractions, so interleaved posterior inference and target updates converge to a unique fixed point.
- Double-sampling EBO removes the overestimation bias of Thompson sampling values in a tabular MDP, while single-sample versions overestimate linearly in noise scale.
- Ensemble-based Epistemic Clipping PPO substantially reduces normalised regret relative to PPO with fixed clipping c=0.2 across several Gymnax environments (20 seeds).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ensemble-based uncertainty-aware adaptive clipping instead of fixed clipping | PPO (Ensemble-ECPPO) · Gymnax discrete-state environments | cumulative regret | Helped |
| Use Laplace-based uncertainty-aware adaptive clipping instead of fixed clipping | PPO (Laplace-ECPPO) · Gymnax discrete-state environments | cumulative regret | Mixed |
| Use Laplace-based uncertainty-aware adaptive clipping instead of fixed clipping | PPO (Laplace-ECPPO) · Breakout-MinAtar | cumulative regret | Hurt |
| Use double sampling to select actions from an independent Q-value sample | Thompson Sampling with EBO · Tabular MDP | value estimation bias | Helped |
| Use double-Q sampling in the MCMC method | Thompson Sampling with MCMC · Tabular MDP | value estimation bias | Helped |
| Use single-sample action selection rather than double sampling (ablation) | Thompson Sampling with EBO · Tabular MDP | value estimation bias | Hurt |
| Use single-sample action selection rather than double-Q sampling (ablation) | Thompson Sampling with MCMC · Tabular MDP | value estimation bias | Hurt |
| Vary the fixed PPO clipping parameter among 0.1, 0.2, and 0.4 (ablation) | PPO · Gymnax discrete-state environments | cumulative regret | Mixed |
Surprising findings
- Laplace-based ECPPO becomes significantly worse on Breakout despite improving several other environments.
Limitations
- Theory relies on tabular or affine settings and Gaussian TD-error noise assumptions that are not generally correct.
- Double-sampling EBO contraction is only conjectured, not proven.
- Laplace-based ECPPO underperforms on some environments such as Breakout.
Experimental setup
Datasets: Gymnax environments OpenAI Gym BSuite MinAtar MNISTBandit-bsuite
Benchmarks: Gymnax discrete environments BSuite MinAtar Tabular MDP Thompson sampling
Metrics: normalised regret value gap to true values Wasserstein distance to fixed point
Baselines: PPO (PureJaxRL) BootDQN EVE Langevin-DQN BDQN LMCDQN SMC-DQN
Builds on: PPO BootDQN EVE Bayesian Bellman Operators Thompson sampling
Closest papers by meaning
- Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning (ICML 2025 · similarity 0.87)
- Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation (ICML 2025 · similarity 0.86)
- Parameterized Projected Bellman Operator (AAAI 2024 · similarity 0.86)
- Bayesian Exploration Networks (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.