Research map / Reinforcement learning / Policy gradient and actor-critic
Off-Policy Evaluation for Action-Dependent Non-stationary Environments
Yash Chandak, Shiv Shankar, Nathaniel D. Bastian, Bruno Castro da Silva, Emma Brunskill, Philip S. Thomas
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OPEN instead of WIS for non-stationary environments | Off-policy evaluation · Non-stationary Mountain Car, Type-1 Diabetes Management, MEDEVAC, and RoboToy | absolute bias and mean-squared error | Improved |
| Use OPEN instead of Pro-WLS in active or hybrid non-stationarity | Off-policy evaluation · Non-stationary Mountain Car, MEDEVAC, and RoboToy | absolute bias and mean-squared error | Improved |
| Use OPEN instead of Pro-WLS in passive non-stationarity | Off-policy evaluation · Type-1 Diabetes Management | absolute bias and mean-squared error | Improved |
| Increase the rate of non-stationarity | WIS · Non-stationary domains | absolute bias | Hurt |
| Use OPEN's autoregressive forecasting rather than Pro-WLS's Fourier-basis regression | Hyperparameter robustness · Non-stationary domains | forecasting robustness | Improved |
| Use importance-weighted instrumental-variable regression instead of naive least-squares regression on noisy performance estimates (ablation) | OPEN · Linear performance-transition model | parameter-estimation consistency | Improved |
| Normalize importance weights on the squared error rather than importance-weighting the regression target (ablation) | OPEN · Linear performance-transition model | parameter-estimation variance | Improved |
| Use a second importance-sampling correction for the effect of behavior-policy actions on the next environment (ablation) | OPEN · Active or hybrid non-stationarity | future-performance prediction | Improved |
| Use importance-weighted regression at high rates of non-stationarity (ablation) | OPEN · Non-stationary domains at high change rates | mean-squared error | Mixed |
Closest papers by meaning
- On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluation (NeurIPS 2024 · similarity 0.87)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.87)
- Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation (NeurIPS 2024 · similarity 0.87)
- Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPs (ICLR 2025 · similarity 0.86)
- Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes (NeurIPS 2024 · similarity 0.86)
- Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
