Research map / Reinforcement learning / Offline RL
EpiCare: A Reinforcement Learning Benchmark for Dynamic Treatment Regimes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EpiCare, a POMDP-based simulated benchmark for longitudinal dynamic treatment regimes, and uses it to evaluate five offline RL methods and five off-policy evaluation estimators.
Problem
Existing RL benchmarks are not medical, and no standardized simulated environment exists for longitudinal dynamic treatment regimes, so offline RL and OPE methods for healthcare cannot be validated against true performance.
Approach
A configurable, procedurally generated POMDP simulator of multi-treatment patient care with hidden disease states, heterogeneous treatment effects, adverse events, and a SMART-style behavior policy that generates offline clinical-trial-like datasets, enabling direct comparison of OPE estimates with true online returns.
Key results
- CQL achieved the best overall performance and, with 131k training episodes, CQL, IQL, DQN, and TD3+BC outperformed the SoC baseline in returns.
- DQN first beat SoC at about 2,048 patients of training data and fell below random below 256 patients, while the paper argues offline RL advantages disappear in data-restricted regimes.
- All five OPE methods (IS, WIS, PDIS, WPDIS, DM) gave poor RMSE estimates of online returns, performing reasonably only for AWAC and BC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train offline RL on SMART clinical-trial data rather than SoC data | Offline RL methods · EpiCare environment 1 | mean return | Mixed |
| Train BC on higher-performing Online data rather than SoC or SMART data | BC · EpiCare environment 1 | mean return: 24.4(1.0) -> 41.5(0.9) -> 66.7(0.8) | Helped |
| Train AWAC on higher-performing Online data rather than SoC or SMART data | AWAC · EpiCare environment 1 | mean return: 30.1(1.8) -> 40.7(1.6) -> 64.4(0.7) | Helped |
| Use SMART rather than SoC training data | CQL · EpiCare environment 1 | mean return: 54.3(0.4) -> 79.4(0.7) | Helped |
| Use SMART rather than SoC training data | DQN · EpiCare environment 1 | mean return: -59.6(7.0) -> 77.0(1.6) | Helped |
| Use SMART rather than SoC training data | IQL · EpiCare environment 1 | mean return: 42.6(0.9) -> 76.5(0.7) | Helped |
| Use SMART rather than SoC training data | TD3+BC · EpiCare environment 1 | mean return: 49.5(1.8) -> 71.3(5.2) | Helped |
| Use SMART rather than Online training data | CQL · EpiCare environment 1 | mean return: 63.9(0.7) -> 79.4(0.7) | Helped |
| Use SMART rather than Online training data | DQN · EpiCare environment 1 | mean return: -32.6(0.7) -> 77.0(1.6) | Helped |
| Use SMART rather than Online training data | IQL · EpiCare environment 1 | mean return: 68.7(0.7) -> 76.5(0.7) | Helped |
| Use SMART rather than Online training data | TD3+BC · EpiCare environment 1 | mean return: 9.6(1.0) -> 71.3(5.2) | Mixed |
| Use AWAC or BC policies for OPE rather than other tested RL models | OPE methods · EpiCare | RMSE against online returns | Mixed |
Surprising findings
- Value-based methods (CQL, DQN, IQL) outperformed actor-critic methods (TD3+BC, AWAC, EDAC) on all metrics averaged across environments.
- Models trained on more exploratory SMART data generally outperformed those trained on higher-performing SoC or Online data.
Limitations
- Results depend on simulation parameters and 8 randomly generated environment seeds; EpiCare is not a model of any specific disease.
- Only five OPE methods tested, list not exhaustive.
- SoC clinician is a modeled heuristic rather than real clinician data.
Experimental setup
Datasets: EpiCare simulated SMART data EpiCare SoC data EpiCare online DQN data
Benchmarks: EpiCare
Metrics: mean return remission rate adverse events per 10k episodes RMSE of OPE estimates vs online returns median returns under data restriction
Baselines: BC DQN SoC policy SMART policy Oracle policy Random policy
Builds on: CORL Gumbel-Softmax TD3+BC CQL IQL
Closest papers by meaning
- Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination (ICML 2024 · similarity 0.92)
- Near-Optimal Reinforcement Learning in Dynamic Treatment Regimes (NeurIPS 2019 · similarity 0.89)
- Designing Optimal Dynamic Treatment Regimes: A Causal Reinforcement Learning Approach (ICML 2020 · similarity 0.87)
- Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies (NeurIPS 2025 · similarity 0.86)
- Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits (AAAI 2025 · similarity 0.85)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.