Research map / Reinforcement learning / Offline RL
Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a closed-form variance-minimizing behavior policy for unbiased Monte Carlo policy evaluation and learns it from offline data via fitted Q-evaluation on a transformed reward, reducing online samples in Gridworld, online stochastic matching, and MuJoCo.
Problem
Online Monte Carlo policy evaluation requires many costly environment interactions; the goal is to reduce estimator variance while remaining unbiased by designing the behavior policy used for data collection.
Approach
Expands the behavior-policy space beyond coverage to a policy space where mu_t(a|s) is proportional to pi_t(a|s) times the square root of an auxiliary second-moment value function; the one-step variant is learned from offline data by applying fitted Q-evaluation twice (first on the original reward, then on a transformed reward 2rq - r^2), avoiding transition model estimation.
Key results
- In Gridworld size 27,000, the proposed method matches on-policy MC error at 750 steps with around 400 steps, saving more than 40% of online interactions.
- In Online Stochastic Matching, variance is reduced by about 42%-49% relative to on-policy Monte Carlo.
- On MuJoCo, the method needs 35-72 episodes versus 100 for on-policy MC to reach the same estimation accuracy (saving up to 65%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ODI behavior-policy design instead of on-policy Monte Carlo | Online stochastic matching · Single- and multi-capacity settings | relative variance: 42%--49% | Helped |
| Use ODI instead of on-policy MC, BPG, or ROS | Gridworld 27,000 · Match estimation accuracy at 300, 600, and 1200 baseline steps | steps: 300 -> 150; 600 -> 330; 1200 -> 540 | Helped |
| Increase ODI offline sample-to-support ratio from 23.15% to 46.3% (ablation) | ODI on Gridworld 27,000 · Match estimation accuracy at 300, 600, and 1200 baseline steps | steps: 150 -> 90; 330 -> 180; 540 -> 420 | Helped |
| Use ODI instead of on-policy MC | MuJoCo policy evaluation · Ant, Hopper, I. D. Pendulum, I. Pendulum, and Walker | episodes: 100 -> 81; 100 -> 54; 100 -> 72; 100 -> 35; 100 -> 70 | Helped |
| Use ODI instead of BPG | MuJoCo policy evaluation · Ant, Hopper, I. D. Pendulum, I. Pendulum, and Walker | episodes: 91 -> 81; 89 -> 54; 103 -> 72; 95 -> 35; 92 -> 70 | Helped |
| Use ODI instead of ROS | MuJoCo policy evaluation · Ant, Hopper, I. D. Pendulum, I. Pendulum, and Walker | episodes: 103 -> 81; 100 -> 54; 99 -> 72; 90 -> 35; 91 -> 70 | Mixed |
Limitations
- Only the one-step optimal policy is learned in practice; the globally optimal policy is not implemented.
- Relies on approximation quality of FQE for the auxiliary value function; theoretical guarantees depend on a relative-accuracy assumption.
- Only PDIS estimator is used; other estimators left for future work.
Experimental setup
Datasets: Gridworld offline data (random actions, uniform state distribution)
Benchmarks: Gridworld Online Stochastic Matching MuJoCo (Ant, Hopper, Inverted Double Pendulum, Inverted Pendulum, Walker)
Metrics: normalized estimation error sample savings (steps or episodes to match on-policy MC) relative variance
Baselines: on-policy Monte Carlo BPG ROS
Builds on: per-decision importance sampling fitted Q evaluation behavior policy gradient (BPG) robust on-policy sampling (ROS)
Closest papers by meaning
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Efficient Multi-Policy Evaluation for Reinforcement Learning (AAAI 2025 · similarity 0.92)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
- Offline Behavior Distillation (NeurIPS 2024 · similarity 0.90)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.90)
- A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.