Research map / Reinforcement learning / Policy gradient and actor-critic
Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes model-free (LSTD-Tournament) and model-based selectors for choosing among candidate OPE value functions or dynamics models, plus a Monte-Carlo, optimization-free experiment protocol demonstrated on Gym Hopper.
Problem
Hyperparameter and model selection for off-policy evaluation (OPE) itself in offline RL, where candidate Q-functions or dynamics models must be selected without introducing further hyperparameters, and where the double-sampling problem biases Bellman-error estimates.
Approach
Builds a tournament selector by creating linear features from pairs of candidate Q-functions so that LSTDQ can check the Bellman equation without helper function classes, achieving n^{-1/2} rates versus BVFT's n^{-1/4}; for model-based selection it constructs helper Bellman-operator classes automatically from the candidate model set, and it evaluates selectors using Monte-Carlo-computed Q-values f
Key results
- LSTD-Tournament is reported as strong and reliable across gravity and noise grids, while the naive model-based selector fails catastrophically in high-noise settings.
- The proposed LSTD-Tournament outperforms the more complex model-based selectors (Zitovsky et al., Antos et al., sign-flip) in the smaller-candidate-set experiments.
- Theory gives an n^{-1/2} rate for LSTD-Tournament versus n^{-1/4} for BVFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LSTD-Tournament instead of BVFT | Model-free OPE selection · Gym Hopper | OPE error | Helped |
| Use a model-based selector instead of LSTD-Tournament | Model-based OPE selection · Gym Hopper, MB.G and MB.N | OPE error | Hurt |
| Select models using next-state prediction loss | Naive model-based selector · Gym Hopper, high-noise MF.N | OPE error | Hurt |
| Replace the standard n^-1/4-rate BVFT approach with LSTD-Tournament | Model-free OPE selection · Theoretical OPE guarantee | sample-rate exponent: n^{-1/4} -> n^{-1/2} | Helped |
| Increase the candidate simulator gap from very small or very large to an intermediate gap | OPE selectors · Gym Hopper noise-grid subgrid study | OPE error | Mixed |
| Increase simulator misspecification | OPE selectors · Gym Hopper noise-grid subgrid study | OPE error | Hurt |
| Increase the fraction of target-policy data in the evaluation dataset | OPE selectors · Gym Hopper data-coverage study | OPE error | Helped |
| Use BVFT rather than TD-square | Model-free OPE selection · Gym Hopper | OPE error | No clear effect |
Surprising findings
- The model-based selectors, despite using more information from the candidate model set, are outperformed by the simpler model-free LSTD-Tournament.
- BVFT's performance mostly coincides with TD-square in the experiments.
Limitations
- Candidate Q-functions generated by varying gravity are structured and may not resemble errors of practical learned FQE/TD functions.
- Theoretical coverage condition via sigma_min(A) is not well understood compared with standard concentrability.
- Model-based selectors scale quadratically with the number of candidate models, limiting them to small candidate sets.
Experimental setup
Datasets: Hopper-v4 simulator-generated offline data (3200 transitions per unit)
Benchmarks: Gym Hopper (Hopper-v4) gravity grid Gym Hopper noise grid
Metrics: OPE error |J_M(pi) - J_M*(pi)|
Baselines: TD-square naive model-based (model prediction loss) BVFT average Bellman error Zitovsky et al. regression selector Antos et al. regression selector
Builds on: BVFT LSTDQ FQE DDPG
Closest papers by meaning
- Off-Policy Evaluation via Off-Policy Classification (NeurIPS 2019 · similarity 0.94)
- Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning (NeurIPS 2021 · similarity 0.93)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- MOPO: Model-based Offline Policy Optimization (NeurIPS 2020 · similarity 0.92)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.