Research map / Reinforcement learning / Policy gradient and actor-critic
Reward Shaping Control Variates for Off-Policy Evaluation Under Sparse Rewards
Ritam Majumdar, Finale Doshi-Velez, Sonali Parbhoo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add learned reward-shaping control variates to PDIS, DR and MRDR | PDIS, DR, MRDR · ICU-Sepsis | variance and MSE: 3-4 orders of magnitude lower variance and MSE | Improved |
| add reward-shaping control variates | traditional OPE estimators · all 11 environments (Chain, cancer, Pointmaze, Antmaze, DOW-30, ICU-Sepsis) | bias, variance, MSE, ESS | Improved |
| add shaping control variates under Gaussian noise on final returns (ablation) | OPE estimators · ICU-Sepsis and DOW-30 noise sweep | estimator error | Improved |
| add shaping control variates as reward signal becomes sparser (ablation) | OPE estimators · ICU-Sepsis and DOW-30 density sweep | estimator error | Improved |
| combine shaping with DR or MRDR instead of PDIS (ablation) | Shaped-PDIS · ICU-Sepsis and DOW-30 sparsity sweep | estimator error | Improved |
| fix lambda = -1 (SCOPE-style uniform shaping) instead of optimizing lambda (ablation) | Shaped-PDIS · theory | variance | Mixed |
| learn potential to maximize explained variance, use variance-optimal lambda | Shaped-PDIS · theory (Corollary 4.5) | variance | Improved |
| learned potential trained for variance reduction rather than as a value function | shaping potential · ICU-Sepsis and DOW-30 Apple | correlation with value function | No clear effect |
Closest papers by meaning
- Automatic Reward Shaping from Confounded Offline Data (ICML 2025 · similarity 0.89)
- Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning (ICLR 2025 · similarity 0.88)
- ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.88)
- Log-Sum-Exponential Estimator for Off-Policy Evaluation and Learning (ICML 2025 · similarity 0.87)
- Policy Evaluation for Variance in Average Reward Reinforcement Learning (ICML 2024 · similarity 0.87)
- Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
