AI Paper Atlas

Research map / Reinforcement learning / Policy gradient and actor-critic

Reward Shaping Control Variates for Off-Policy Evaluation Under Sparse Rewards

Ritam Majumdar, Finale Doshi-Velez, Sonali Parbhoo

ICML 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Importance-weighted OPE estimators

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add learned reward-shaping control variates to PDIS, DR and MRDRPDIS, DR, MRDR · ICU-Sepsisvariance and MSE: 3-4 orders of magnitude lower variance and MSEImproved
add reward-shaping control variatestraditional OPE estimators · all 11 environments (Chain, cancer, Pointmaze, Antmaze, DOW-30, ICU-Sepsis)bias, variance, MSE, ESSImproved
add shaping control variates under Gaussian noise on final returns (ablation)OPE estimators · ICU-Sepsis and DOW-30 noise sweepestimator errorImproved
add shaping control variates as reward signal becomes sparser (ablation)OPE estimators · ICU-Sepsis and DOW-30 density sweepestimator errorImproved
combine shaping with DR or MRDR instead of PDIS (ablation)Shaped-PDIS · ICU-Sepsis and DOW-30 sparsity sweepestimator errorImproved
fix lambda = -1 (SCOPE-style uniform shaping) instead of optimizing lambda (ablation)Shaped-PDIS · theoryvarianceMixed
learn potential to maximize explained variance, use variance-optimal lambdaShaped-PDIS · theory (Corollary 4.5)varianceImproved
learned potential trained for variance reduction rather than as a value functionshaping potential · ICU-Sepsis and DOW-30 Applecorrelation with value functionNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy