Research map / Reinforcement learning / Policy gradient and actor-critic
Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SharpeRatio@k, a finance-inspired metric that scores an off-policy evaluation estimator by the risk-return tradeoff of its top-k policy portfolio, and benchmarks common OPE estimators on MuJoCo and classic control tasks using the SCOPE-RL library.
Problem
Existing evaluation-of-OPE metrics (MSE, rank correlation, regret@k) measure accuracy of estimation or top-1 policy selection but ignore the risk of deploying suboptimal policies in the subsequent online A/B test stage, where the top-k shortlisted policies are evaluated.
Approach
Treat the top-k policies chosen by an OPE estimator as a "policy portfolio" and score it with a Sharpe-ratio analogue: (best@k minus behavior policy value) divided by the standard deviation of policy values among the top-k, so that both return and risk of the shortlist are captured across online budgets k.
Key results
- SharpeRatio@5 has Spearman rank correlation with RankCorr above 0.7 in Reacher (0.81), Hopper (0.70), and Swimmer (0.79), but only 0.18 in InvertedPendulum.
- SharpeRatio@5 and RankCorr disagree on the best estimator in 36 of 70 trials across 7 tasks and 10 seeds; nRegret disagrees in 59 of 70.
- In MountainCar, MSE and Regret select MIS, RankCorr and SharpeRatio@7 favor DM, and SharpeRatio@4 selects PDIS, showing the best estimator depends on the metric and budget k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate OPE estimators with SharpeRatio@k instead of relying only on conventional accuracy metrics | OPE estimator selection · MountainCar | estimator selection | Mixed |
| Use SharpeRatio@k to account for portfolio risk as well as return | OPE estimator selection · MountainCar | risk-return efficiency | Helped |
| Select MIS rather than DM using SharpeRatio@k | OPE estimators · Hopper | SharpeRatio@k ranking | Helped |
| Use SharpeRatio@k across different online evaluation budgets | OPE estimator evaluation · Most benchmark environments | SharpeRatio@k evaluation consistency | Helped |
| Use online evaluation budgets k ≤ 3 | OPE estimator evaluation · Benchmark environments | SharpeRatio@k stability | Hurt |
| Use SharpeRatio@k rather than RankCorr for choosing an estimator | OPE estimator selection · MountainCar | risk-aware estimator ranking | Helped |
Surprising findings
- DR and MDR do not consistently outperform DM, PDIS, and MIS under SharpeRatio@k, despite being regarded as advanced estimators.
- A low-variance estimator (DM) can rank worst under SharpeRatio@k while MIS with similar std ranks best, because DM overestimates poor policies.
Limitations
- Benchmarks use only 7 small classic-control and MuJoCo tasks with a fixed set of 10 candidate policies per trial.
- Uses true (self-normalized) importance weights and short max episode lengths to avoid extreme weights, which deviates from practical OPE settings.
- Hyperparameters of FQE and DICE were not extensively tuned.
Experimental setup
Datasets: Gym-Mujoco Reacher Gym-Mujoco InvertedPendulum Gym-Mujoco Hopper Gym-Mujoco Swimmer Gym-Classic Control CartPole Gym-Classic Control MountainCar Gym-Classic Control Acrobot
Benchmarks: Reacher InvertedPendulum Hopper Swimmer CartPole MountainCar Acrobot
Metrics: SharpeRatio@k nMSE RankCorr nRegret@1 std@k best@k
Baselines: MSE rank correlation regret@k DOPE COBS
Builds on: Sharpe ratio fitted Q evaluation doubly robust off-policy evaluation DOPE COBS
Closest papers by meaning
- Policy-Adaptive Estimator Selection for Off-Policy Evaluation (AAAI 2023 · similarity 0.89)
- Universal Off-Policy Evaluation (NeurIPS 2021 · similarity 0.88)
- Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation (NeurIPS 2017 · similarity 0.87)
- Benchmarks for Deep Off-Policy Evaluation (ICLR 2021 · similarity 0.87)
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.86)
- Off-Policy Evaluation under Nonignorable Missing Data (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.