Research map / Reinforcement learning / Policy gradient and actor-critic
Importance-weighted OPE estimators
Build doubly robust and marginalized importance-sampling estimators for off-policy evaluation.
57 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Policy gradient and actor-critic, keyword cluster “ope · estimators · importance”
State History Tried Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
13 papers in 2023–24 and 9 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 3.8% to 1.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (ICML 2016 · 79 citations)
- 2017 · Off-policy evaluation for slate recommendation (NeurIPS 2017 · 39 citations)
- 2018 · Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation (NeurIPS 2018 · 94 citations)
- 2019 · CAB: Continuous Adaptive Blending for Policy Evaluation and Learning (ICML 2019 · 32 citations)
- 2020 · Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (ICLR 2020 · 29 citations)
- 2021 · Bootstrapping Fitted Q-Evaluation for Off-Policy Inference (ICML 2021 · 10 citations)
- 2022 · Towards Robust Off-Policy Learning for Runtime Uncertainty (AAAI 2022 · 2 citations)
- 2023 · Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction (AAAI 2023 · 2 citations)
- 2024 · Distributional Off-Policy Evaluation for Slate Recommendations (AAAI 2024 · 2 citations)
- 2025 · Cross-Validated Off-Policy Evaluation (AAAI 2025 · 2 citations)
- 2026 · Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning (AAAI 2026 · 0 citations)
Earliest papers here
- Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (ICML 2016 · 79 citations)
- Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (ICML 2016 · 73 citations)
- Off-policy evaluation for slate recommendation (NeurIPS 2017 · 39 citations)
What has been tried
50 papers here report 418 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 8%, hurt 19%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add the one-standard-error rule to standard 10-fold OCV-DR with a 90/10 split | Average across all datasets · MSE | Helped | Cross-Validated Off-Policy Evaluation (AAAI 2025 · 2 citations) | |
| add behavior-policy cross-entropy loss to transition-only training | Simulated dynamic process, policies A and B · LMSE | 4.72 -> 4.23; 5.14 -> 4.47 | Helped | Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning (NeurIPS 2024 · 2 citations) |
| Increase empirical moment order from L=1 to L>1 using the optimal regularizer | offline contextual bandit risk evaluation · upper-bound tightness | Helped | Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning (NeurIPS 2024 · 2 citations) | |
| add learned kernel metric | Pendulum with dummy dim · RMSE | 0.250 ± 0.032 -> 0.115 ± 0.018 | Helped | Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies (ICLR 2024 · 1 citation) |
| Add the action-advantage correction (DAE) to Uncorrected | Deterministic MinAtar environments · policy score | Helped | Skill or Luck? Return Decomposition via Advantage Functions (ICLR 2024 · 0 citations) |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 22 papers; from OpenAlex and the papers' affiliations).
- University of Wisconsin - Madison · 2 papers
- Adobe · 2 papers
- University of Massachusetts Amherst · 2 papers
- Ecole Nationale de la Statistique et de l'Administration Economique · 2 papers
- London School of Economics · 2 papers
- Tsinghua University · 2 papers
Use it
Download BibTeX of 35 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Confounding-robust policy evaluation (47 papers)
- Confidence intervals and OPE benchmarks (28 papers)
Elsewhere in the topic
- Model-based RL with guarantees (245 papers)
- Regularized scalable deep value learning (216 papers)
- Temporal-difference and discounting theory (143 papers)
- Tree search and policy optimization (109 papers)
- Safety layers and shielding (85 papers)
- Variance-reduced policy gradients (79 papers)
- Actor-critic theory (76 papers)
- Off-policy actor-critic with expressive policies (71 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).