Research map / Reinforcement learning / Offline RL
OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OPERA, a meta-algorithm that combines multiple off-policy evaluation estimators via a linear weighted stack whose weights are chosen by minimizing a bootstrap-estimated MSE, with finite-sample and consistency analysis.
Problem
Offline policy evaluation requires choosing among many OPE estimators with hyperparameters and no reliable way to pick the best one per task and domain.
Approach
Treat OPE estimates as inputs to a linear stack whose weights minimize a plug-in MSE matrix estimated by subsampled bootstrap, which avoids needing ground-truth values or structural assumptions on the estimators (no nonnegativity or nested bias ordering required).
Key results
- On D4RL Hopper medium, OPERA achieves RMSE 8.5 versus 12.5 for BestOPE and 16.4 for BVFT.
- On Sepsis MDP with N=200, OPERA achieves MSE 0.2205 versus 0.2448 for the best single estimator (FQE).
- On ToyGraph deterministic MDP, OPERA achieves MSE 0.0339 versus 0.0509 for BestOPE and WIS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine OPE estimators with OPERA | Sepsis policy evaluation · Sepsis-MDP N=200 | MSE: 0.2448 -> 0.2205 | Helped |
| Combine OPE estimators with OPERA | Sepsis policy evaluation · Sepsis-MDP N=1000 and Sepsis-POMDP | MSE: 0.1720 -> 0.1705; 0.2804 -> 0.2750; 0.2799 -> 0.2749 | Helped |
| Combine OPE estimates with OPERA | D4RL policy evaluation · Hopper, HalfCheetah, and Walker2d medium and medium-replay | RMSE: 15.5 -> 13.0; 12.5 -> 8.5; 65.0 -> 46.0; 111.8 -> 100.5; 167.4 -> 138.3; 183.8 -> 149.0 | Helped |
| Combine OPE estimates with OPERA | Graph policy evaluation · Graph deterministic and stochastic MDP; stochastic POMDP | MSE: 0.0509 -> 0.0339; 0.4838 -> 0.4625; 0.3516 -> 0.3327 | Helped |
| Combine OPE estimates with OPERA | Graph policy evaluation · Graph deterministic POMDP | MSE: 1.2193 -> 1.4651 | Hurt |
| Use IS as the centering estimator (ablation) | OPERA · Sepsis-MDP N=200 and Sepsis-POMDP N=1000 | MSE: 0.2205 -> 0.2181; 0.2749 -> 0.2720 | Helped |
| Use IS as the centering estimator (ablation) | OPERA · Sepsis-MDP N=1000 and Sepsis-POMDP N=200 | MSE: 0.1705 -> 0.1779; 0.2750 -> 0.2768 | Hurt |
| Estimate bias with MAGIC's WIS confidence-bound method (ablation) | OPERA · Sepsis-MDP and Sepsis-POMDP | MSE: 0.2205 -> 0.2657; 0.1705 -> 0.1848; 0.2750 -> 0.2827; 0.2749 -> 0.2802 | Hurt |
Surprising findings
- OPERA is worse than BestOPE on the deterministic POMDP ToyGraph setting, attributed to bootstrap MSE estimation error.
- OPERA-MAGIC, which uses MAGIC's bias estimate, almost always performs worse than the best estimator in the ensemble.
Limitations
- In the ultra-small data regime OPERA underperforms single-estimator selection methods because it does not explicitly reject bad estimators.
- Consistency and performance guarantees rely on regularity assumptions (bounded, smooth, consistent base estimators) that may not hold for FQE with function approximation.
- For a particular policy, OPERA may not outperform the best individual estimator; gains are shown mainly when averaging over many policies.
Experimental setup
Datasets: D4RL Hopper medium-replay D4RL Hopper medium D4RL HalfCheetah medium-replay D4RL HalfCheetah medium D4RL Walker2d medium-replay D4RL Walker2d medium Sepsis simulator ToyGraph SLOPE contextual bandit synthetic domain
Benchmarks: D4RL Sepsis simulator ToyGraph contextual bandit (SLOPE domain)
Metrics: mean squared error root mean squared error normalized MSE CDF
Baselines: AvgOPE BestOPE BVFT SLOPE DR IS WIS FQE DualDICE MB
Builds on: fitted Q evaluation MAGIC SLOPE BVFT stacked generalization
Closest papers by meaning
- Benchmarks for Deep Off-Policy Evaluation (ICLR 2021 · similarity 0.94)
- STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation (NeurIPS 2025 · similarity 0.88)
- Off-Policy Evaluation under Nonignorable Missing Data (ICML 2025 · similarity 0.88)
- Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation (NeurIPS 2026 · similarity 0.87)
- Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation (NeurIPS 2024 · similarity 0.87)
- Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.