Research map / Reinforcement learning / Offline RL
Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing offline RL for dynamic treatment regimes needs reassessment, supported by a sepsis case study (17,550 evaluation experiments) showing RL rankings shift with OPE metrics and reward design and that naive baselines can win.
Problem
Offline RL for dynamic treatment regimes (DTRs) lacks standardized policy evaluation metrics, supervised and naive baselines, and consistent MDP/reward formulations, making it unclear whether RL truly improves treatment decisions.
Approach
Systematically varies OPE estimators (WIS variants, DR), reward designs (outcome, SOFA-based, NEWS2-based), and baselines (random, min/max/alternating, weighted behavior policy, LSTM supervised learning) on a MIMIC-III sepsis cohort to show RL rankings are unstable and that naive baselines can outperform RL and clinicians under OPE.
Key results
- Across 10 policies, 9 metrics, 13 patient groups and 3 reward designs, the authors report 17,550 evaluation experiments; the best RL algorithm changes with reward (DQN wins most on NEWS2, CQL most on outcome reward).
- On the NEWS2 test set, the naive weight policy scores -3.78 under WIS, beating all RL, SL and the clinician return (-4.39), illustrating naive baselines winning under OPE.
- Temperature-scaling calibration of the behavior model increases extreme importance ratios, and some naive baselines still surpass clinician returns, suggesting calibration does not reliably fix OPE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use different reward designs (outcome, SOFA, or NEWS2) | Offline RL algorithms on the Sepsis dataset · Sepsis test set | Number of wins | Mixed |
| Compare RL policies with naive policies using OPE | Offline RL evaluation on the Sepsis dataset · Sepsis test set | WIS: -3.78 vs -3.79 | Mixed |
| Compare RL with supervised learning across metrics | Offline RL algorithms on the Sepsis dataset · Sepsis test set and patient subgroups | Performance across OPE and supervised-learning metrics | Mixed |
| Apply temperature scaling to the behavior model | OPE for naive policies on the Sepsis dataset · Sepsis test set | OPE variance | Hurt |
| Evaluate policies separately by NEWS2 risk-change subgroup | Offline RL policies on the Sepsis dataset · Patient subgroups | Performance | Mixed |
| Use CQL rather than SL | Sepsis policies · Specific patient subgroups | OPE and supervised-learning metrics | Helped |
Surprising findings
- Naive baselines (e.g., a frequency-matching weight policy) outperform RL, SL and clinician returns on several OPE metrics, which the authors treat as a sign that OPE is unreliable.
- Doubly robust estimates often deviate from IS estimates in both directions, so DR does not reliably behave as doubly robust in this setting.
Limitations
- Single sepsis dataset (MIMIC-III), so generalizability to other DTR datasets is untested.
- Linear function approximation only; no exploration of neural or recurrent representations.
- Does not model multiple optimal treatments per patient.
Experimental setup
Datasets: MIMIC-III sepsis cohort (derived following AI Clinician preprocessing)
Benchmarks: MIMIC-III sepsis cohort test set
Metrics: WIS WIS_b WIS_t WIS_bt DR RMSE for IV fluid and vasopressor patient-wise F1 sample-wise F1 number of wins
Baselines: DQN CQL BCQ IQL LSTM supervised learning random policy min policy max policy alternating policy weight policy (behavior frequency) clinician policy
Builds on: DQN Conservative Q-Learning Batch-Constrained Q-learning Implicit Q-Learning doubly robust off-policy evaluation
Closest papers by meaning
- EpiCare: A Reinforcement Learning Benchmark for Dynamic Treatment Regimes (NeurIPS 2024 · similarity 0.92)
- Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies (NeurIPS 2025 · similarity 0.88)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.86)
- Evaluating and Learning Optimal Dynamic Treatment Regimes under Truncation by Death (NeurIPS 2025 · similarity 0.86)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.