Research map / Reinforcement learning / Policy gradient and actor-critic
Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCOPE, a model-free off-policy evaluation method that learns a variance-minimizing behavior policy under a cost constraint relative to the target policy, using offline data and convex optimization.
Problem
On-policy policy evaluation in RL has high variance and needs much online data, while existing variance-reducing behavior policies ignore execution safety (cost constraints).
Approach
Derives a recursive extended reward so that the trajectory variance of the PDIS estimator reduces to a one-step contextual bandit problem per time step; solves a convex program per step for a behavior policy that minimizes variance subject to an expected-cost threshold (1+epsilon) times the target policy's cost, learned from offline data via FQE.
Key results
- On Gridworld (1,000 states), SCOPE's average trajectory cost is 0.861 of on-policy cost, while ODI costs 1.602 and ROS 1.083.
- SCOPE reaches on-policy accuracy with 425 cost vs 1000 for on-policy on InvertedPendulum (57.5% saved), and saves 25.4% to 57.5% across five MuJoCo environments.
- Relative variance on Gridworld: SCOPE 0.547 (1,000 states) and 0.575 (27,000 states) vs ODI 0.460 and 0.484, so ODI achieves slightly lower variance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCOPE's safety-constrained behavior policy instead of On-policy MC | Policy evaluation · Gridworld (1,000 states) | relative variance: 1.000 -> 0.547 | Helped |
| Use SCOPE's safety-constrained behavior policy instead of On-policy MC | Policy evaluation · Gridworld (27,000 states) | relative variance: 1.000 -> 0.575 | Helped |
| Use SCOPE instead of ODI | Policy evaluation · Gridworld (1,000 states) | relative variance: 0.460 -> 0.547 | Hurt |
| Use SCOPE instead of ODI | Policy evaluation · Gridworld (1,000 states) | average trajectory cost (normalized): 1.602 -> 0.861 | Helped |
| Use SCOPE instead of On-policy MC | Policy evaluation · Gridworld (1,000 states) | average trajectory cost (normalized): 1.000 -> 0.861 | Helped |
| Use SCOPE instead of On-policy MC | Policy evaluation · Gridworld (27,000 states) | average trajectory cost (normalized): 1.000 -> 0.849 | Helped |
| Use SCOPE instead of ODI | Policy evaluation · Gridworld (27,000 states) | average trajectory cost (normalized): 1.590 -> 0.849 | Helped |
| Use SCOPE instead of On-policy MC | Policy evaluation · Gridworld (10) | cost to match On-policy MC accuracy at 1000 episodes: 1000 -> 472 | Helped |
| Use SCOPE instead of On-policy MC | Policy evaluation · Gridworld (30) | cost to match On-policy MC accuracy at 1000 episodes: 1000 -> 487 | Helped |
| Use SCOPE instead of On-policy MC | Policy evaluation · Ant | cost to match On-policy MC accuracy at 1000 episodes: 1000 -> 746 | Helped |
| Use SCOPE instead of On-policy MC | Policy evaluation · Hopper | cost to match On-policy MC accuracy at 1000 episodes: 1000 -> 552 | Helped |
| Use SCOPE instead of On-policy MC | Policy evaluation · InvertedDoublePendulum | cost to match On-policy MC accuracy at 1000 episodes: 1000 -> 681 | Helped |
Surprising findings
- ODI achieves slightly lower variance than SCOPE on Gridworld but at much higher cost (more than 1.5x on-policy), showing variance-only methods trade off safety heavily.
Limitations
- Performance may be inaccurate when offline data is very small (e.g., a single tuple), where the paper recommends on-policy evaluation.
- Analysis restricted to Monte Carlo (no temporal-difference learning); TD extension left to future work.
- Safety constraint is relative to target policy with epsilon=0 used in experiments, not an absolute safety threshold.
Experimental setup
Datasets: Gridworld (synthetic, n^3 states) MuJoCo offline logged data (Ant, Hopper, InvertedDoublePendulum, InvertedPendulum, Walker)
Benchmarks: Gridworld (1,000 and 27,000 states) MuJoCo (Ant, Hopper, InvertedDoublePendulum, InvertedPendulum, Walker)
Metrics: estimation error normalized by on-policy error variance total cost to reach target accuracy relative variance
Baselines: On-policy Monte Carlo ROS ODI
Builds on: per-decision importance sampling fitted q evaluation robust on-policy sampling offline data informed estimator
Closest papers by meaning
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.90)
- Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation (AAAI 2024 · similarity 0.90)
- Efficient Multi-Policy Evaluation for Reinforcement Learning (AAAI 2025 · similarity 0.90)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.89)
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.89)
- Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.