Research map / Reinforcement learning / Policy gradient and actor-critic
Cross-Validated Off-Policy Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes off-policy cross-validation (OCV), which uses repeated train/validation splits of a single logged dataset with an unbiased validator (IPS or DR) to select among OPE estimators and tune their hyper-parameters.
Problem
Estimator selection and hyper-parameter tuning in off-policy evaluation, where theory gives limited guidance and the true policy value is unknown, so standard cross-validation is assumed infeasible.
Approach
Use an unbiased off-policy estimator (IPS or DR) on held-out validation splits as a validator, measure squared loss between evaluated estimators and the validator across repeated random splits, choose the estimator with lowest one-standard-error upper bound, and set the train/validation ratio from estimated variances.
Key results
- OCV with DR validator (OCV-DR) significantly outperforms SLOPE and PAS-IF on two datasets while never being much worse in the IPS/DM/DR selection experiment.
- Across 90 conditions for hyper-parameter tuning of seven estimators, OCV often matches theory-suggested hyper-parameters and typically outperforms SLOPE and PAS-IF.
- OCV is about 100 times less computationally costly than PAS-IF (0.06-0.13s vs 13.91s per evaluation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cross-validated estimator selection instead of Slope or PAS-IF | OPE estimator selection among IPS, DM, and DR · Nine UCI datasets, target temperature β₁ = 10 | MSE | Helped |
| Use OCV-DR instead of the selection baselines | OPE estimator selection among IPS, DM, and DR · Two datasets, target temperature β₁ = 10 | MSE | Helped |
| Use cross-validated estimator selection instead of Slope or PAS-IF | OPE estimator selection among IPS, DM, and DR · Nine UCI datasets, target temperature β₁ = -10 | MSE | Helped |
| Use OCV for hyper-parameter tuning | Seven OPE estimators · 90 conditions across nine UCI datasets | MSE | Helped |
| Use OCV-IPS or OCV-DR for joint estimator selection and hyper-parameter tuning | Joint estimator and hyper-parameter selection · Everything | MSE: an order of magnitude lower MSE | Helped |
| Use OCV instead of PAS-IF | Policy evaluation with cross-validation · Single policy evaluation | time: 100 times less computationally costly | Helped |
| Add the one-standard-error rule to standard 10-fold OCV-DR with a 90/10 split (ablation) | OCV-DR estimator selection · Average across all datasets | MSE | Helped |
| Use the theory-driven adaptive training/validation split ratio instead of a 90/10 split (ablation) | OCV-DR estimator selection · Average across all datasets | MSE | Helped |
| Combine the theory-driven split ratio with the one-standard-error rule (ablation) | OCV-DR estimator selection · Average across all datasets | MSE | No clear effect |
| Increase the number of training/validation splits from the combined method's setting to K=100 (ablation) | OCV-DR estimator selection · Average across all datasets | MSE: additional marginal improvements | Helped |
| Increase the number of repeated training/validation splits (ablation) | OCV-DR · Vehicle | MSE | Helped |
| Use DM as the cross-validation validator instead of an unbiased validator (ablation) | OCV estimator selection · Estimator selection among IPS, DM, and DR | MSE | Hurt |
Surprising findings
- Using a biased DM validator makes cross-validation prefer DM itself even when DM performs poorly.
- PAS-IF tends to prefer DM despite DM performing poorly, which the authors attribute to its biased validation procedure.
Limitations
- Validator must be unbiased; biased validators such as DM systematically favor biased estimators.
- Off-policy learning and reinforcement learning extensions are left as future work.
- Gains over theory-driven tuning are not uniform; OCV often only matches theory-suggested values.
Experimental setup
Datasets: ecoli glass letter optdigits page-blocks pendigits satimage vehicle yeast
Benchmarks: nine OpenML classification datasets converted to contextual bandits
Metrics: mean squared error regret
Baselines: SLOPE PAS-IF theory-suggested hyper-parameter values OCV with DM validator
Builds on: inverse propensity scoring doubly robust estimator Monte Carlo cross-validation one standard error rule SLOPE PAS-IF
Closest papers by meaning
- Off-policy Evaluation Beyond Overlap: Sharp Partial Identification Under Smoothness (ICML 2024 · similarity 0.89)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.89)
- More Efficient Off-Policy Evaluation through Regularized Targeted Learning (ICML 2019 · similarity 0.87)
- Supervised Off-Policy Ranking (ICML 2022 · similarity 0.87)
- Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation (NeurIPS 2026 · similarity 0.86)
- Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.