Research map / Reinforcement learning / Offline RL
Selective Uncertainty Propagation in Offline RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes selective uncertainty propagation for confidence intervals on the step-wise treatment effect in finite-horizon offline RL, and a selectively pessimistic value iteration (SPVI) that interpolates between contextual bandit and full RL pessimism based on estimated next-state shift.
Problem
Offline RL in the finite-horizon setting requires learning the policy at each DP step, which needs confidence intervals for the treatment effect of deviating from the behavioral policy; standard pessimism propagates uncertainty from future steps even when actions barely affect next-state distributio
Approach
Scale the per-state pointwise uncertainty of next-step value estimates by the estimated action-induced next-state shift, so uncertainty is propagated only when actions affect transitions; when the shift is zero the bound reduces to contextual-bandit-style guarantees independent of state-space size.
Key results
- On ChainBandit, selective propagation gives tighter CIs for the step-2 treatment effect than standard propagation, with the benefit largest when the evaluation policy is close to the behavioral policy.
- SPVI is reported as the best-performing policy learner on all ChainBandit simulations tried, eventually outperforming PVI and PSL despite a worse start.
- On the dynamic GridWorld, selective and standard propagation perform similarly, as expected since most actions change next-state distributions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use selective uncertainty propagation instead of standard propagation | Offline policy learning on ChainBandit · ChainBandit | policy value | Helped |
| use selective uncertainty propagation instead of standard propagation | Offline policy learning on ChainBandit · ChainBandit | policy value | Mixed |
| omit planning and use pessimistic supervised learning | Offline policy learning on ChainBandit · ChainBandit | policy value | Hurt |
| use selective rather than standard uncertainty propagation | Confidence interval construction · ChainBandit | confidence interval width | Helped |
| use selective rather than standard uncertainty propagation | Confidence interval construction · GridWorld | confidence interval width | No clear effect |
| use selective rather than standard uncertainty propagation | Offline policy learning · GridWorld | policy value | No clear effect |
| omit planning and use pessimistic supervised learning | Offline policy learning on GridWorld · GridWorld | policy value | Hurt |
Surprising findings
- SPVI starts worse than PVI under a highly suboptimal behavioral policy, because selective pessimism initially biases against policies that induce large shifts.
Limitations
- Theory assumes access to externally provided CB, RL value, and shift estimates with high-probability bounds; the shift estimator is not developed in the paper.
- Empirical evaluation is limited to small tabular toy environments (ChainBandit and GridWorld).
- Under dynamic environments like GridWorld, selective propagation offers no advantage over standard propagation.
Experimental setup
Datasets: ChainBandit (simulated) GridWorld (simulated)
Benchmarks: ChainBandit GridWorld
Metrics: confidence interval width and validity for alpha^h learned policy value vs number of training episodes
Baselines: pessimistic value iteration (PVI) pessimistic supervised learning (PSL) standard uncertainty propagation
Builds on: pessimistic value iteration
Closest papers by meaning
- Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning (ICML 2026 · similarity 0.91)
- Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.91)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.