Research map / Reinforcement learning / Policy gradient and actor-critic
Efficient Multi-Policy Evaluation for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Designs a tailored shared behavior policy (MPE) that unbiasedly evaluates multiple target policies with lower variance than on-policy Monte Carlo, using a similarity condition so sample needs do not scale with the number of policies.
Problem
Efficiently and unbiasedly estimating the performance of multiple target policies in RL, where on-policy evaluation needs samples scaling with the number of policies and prior off-policy reuse methods rely on restrictive assumptions or impractical knowledge.
Approach
Enlarges the behavior-policy search space to any policy whose support covers only actions with nonzero pi(a)q(a) (preserving unbiasedness for PDIS), derives a closed-form optimal behavior policy mu(a) proportional to sqrt(sum_k pi_k(a)^2 q_k(a)^2) (extended to RL via a learned second-moment q-hat), and shows that sharing its samples across K target policies lowers variance under a policy-similarit
Key results
- On Gridworld, the proposed estimator reduces variance to about 12.5% of on-policy Monte Carlo (0.125 and 0.129 relative variance at 1,000 and 27,000 states).
- Matching on-policy MC accuracy at 1,000 episodes needs only about 126-131 episodes on Gridworld, saving about 87% of online interactions.
- On MuJoCo, the estimator reduces variance to about 10% of on-policy Monte Carlo and saves about 90% of online interactions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MPE's tailored shared behavior policy instead of on-policy MC | Multi-policy evaluation · Gridworld 1,000 states | relative variance: 1.000 -> 0.125 | Helped |
| Use MPE's tailored shared behavior policy instead of on-policy MC | Multi-policy evaluation · Gridworld 27,000 states | relative variance: 1.000 -> 0.129 | Helped |
| Use MPE's tailored shared behavior policy instead of ODI | Multi-policy evaluation · Gridworld 1,000 states | relative variance: 0.637 -> 0.125 | Helped |
| Use MPE's tailored shared behavior policy instead of ODI | Multi-policy evaluation · Gridworld 27,000 states | relative variance: 0.601 -> 0.129 | Helped |
| Use MPE's tailored shared behavior policy instead of on-policy MC | Multi-policy evaluation · Gridworld 1,000 states | episodes to match on-policy MC accuracy: 1000 -> 126 | Helped |
| Use MPE's tailored shared behavior policy instead of on-policy MC | Multi-policy evaluation · Gridworld 27,000 states | episodes to match on-policy MC accuracy: 1000 -> 131 | Helped |
| Reuse samples across target policies without a jointly tailored behavior policy (SON) | Multi-policy evaluation · MuJoCo | variance | Hurt |
| Reuse samples from separately tailored ODI behavior policies (SODI) | Multi-policy evaluation · MuJoCo | variance | Hurt |
Surprising findings
- Naive sample sharing (SON and SODI) performs worse than on-policy Monte Carlo when target policies are dissimilar.
- SODI, which tailors a behavior policy per target policy, performs worse than SON, suggesting per-policy tailored behaviors are vulnerable to target policy change.
Limitations
- Variance reduction depends on similarity among target policies, and the condition must be checked.
- The method relies on learning the second-moment quantity q-hat with offline RL, whose approximation error is not analyzed in the main experiments.
- Only the PDIS estimator is studied; other importance-sampling estimators are left for future work.
Experimental setup
Datasets: Gridworld MuJoCo
Benchmarks: Gridworld MuJoCo
Metrics: relative variance episodes needed for equal estimation accuracy relative estimation error
Baselines: on-policy Monte Carlo ODI SON SODI
Builds on: per-decision importance sampling PPO offline data informed behavior policy (ODI) fitted Q evaluation
Closest papers by meaning
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.92)
- Doubly Optimal Policy Evaluation for Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation (ICML 2025 · similarity 0.88)
- Generalizable Policy Improvement via Reinforcement Sampling (Student Abstract) (AAAI 2024 · similarity 0.88)
- Constructing an Optimal Behavior Basis for the Option Keyboard (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.