Research map / Reinforcement learning / Offline RL
Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes delphic uncertainty, the variance of counterfactual value estimates across compatible confounded world models, and uses it as a pessimism penalty in CQL to improve offline RL under nonidentifiable hidden confounding.
Problem
Offline RL with nonidentifiable hidden confounding, where unobserved context affects both behaviour actions and outcomes, so value estimates for context-independent policies are biased and cannot be fixed by standard statistical uncertainty methods.
Approach
Define delphic uncertainty as the variance over world models (confounder priors, behaviour policies, value functions) that all induce the same observational trajectory distribution, decompose total variance into aleatoric, epistemic and delphic terms, and penalise the Bellman target of CQL with this variance.
Key results
- On the sepsis simulator, delphic ORL outperforms CQL, BCQ and BC especially as confounding strength Γ increases, with results averaged over 10 runs with 95% CIs.
- Delphic uncertainty increases with confounding strength and is highest under vasopressor administration, the only treatment whose outcomes are confounded by the hidden diabetic status, while it does not decrease with more data.
- On real ICU data with artificially excluded confounders (Γ up to 200), delphic ORL maintains improved doubly-robust OPE performance where offline RL baselines degrade; six expert clinicians also preferred its policies in pairwise rankings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| penalize Q-values using delphic uncertainty | Delphic ORL (CQL) · Sepsis simulation | normalized environment return | Helped |
| penalize Q-values using delphic uncertainty | Delphic ORL · Real-world medical dataset | off-policy evaluation | Helped |
| use behaviour cloning instead of offline RL | Offline RL policies · Sepsis simulation | normalized environment return | Mixed |
| learn a policy with delphic-uncertainty pessimism rather than behaviour cloning (ablation) | Delphic ORL · Real-world medical dataset | policy action discrepancy from the data policy | Mixed |
| increase the number of compatible world models (ablation) | Delphic uncertainty estimation · Sepsis simulation and real-world medical dataset | delphic uncertainty | No clear effect |
Surprising findings
- Delphic uncertainty does not decrease with more data, unlike epistemic uncertainty, even in deterministic settings.
- Behaviour cloning is less prone to confounding bias than offline RL methods in the sepsis experiments, yet still underperforms delphic ORL under strong confounding.
Limitations
- Empirical evaluation limited to medically motivated confounding scenarios, with artificial confounders on the ICU data.
- Computational cost of training an ensemble of compatible world models can be high for large, highly confounded problems.
- Quality of the approximation to the set of compatible worlds is not theoretically guaranteed, and hyperparameters (number of worlds, lambda) require tuning.
Experimental setup
Datasets: sepsis simulator (adapted from prior work) public ICU electronic health records dataset (over 33k patient stays, 200+ variables)
Benchmarks: sepsis simulation real-world ICU electronic health records
Metrics: normalised environment return off-policy evaluation value (doubly robust) delphic uncertainty magnitude expert clinician pairwise ranking
Baselines: Conservative Q-Learning Batch-Constrained Q-Learning behaviour cloning
Builds on: Conservative Q-Learning contextual MDP variational inference (ELBO) law of total variance
Closest papers by meaning
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.90)
- Selective Uncertainty Propagation in Offline RL (AAAI 2025 · similarity 0.89)
- Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.89)
- Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.89)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.