Research map / Reinforcement learning / Policy gradient and actor-critic

Doubly Optimal Policy Evaluation for Reinforcement Learning

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Confounding-robust policy evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Doubly Optimal (DOpt) policy evaluation that jointly derives a closed-form optimal behavior policy and optimal baseline (q-function) for PDIS-based off-policy evaluation, showing unbiasedness and lower variance than on-policy MC, ODI and DR estimators.

Problem

Policy evaluation in finite-horizon RL suffers high variance over long horizons, requiring many online samples; existing behavior-policy design methods ignore the data-processing baseline, and baseline methods ignore behavior-policy design.

Approach

Solve a bi-level problem: for any baseline, the variance-optimal behavior policy is mu*_t(a|s) proportional to pi_t(a|s) sqrt(u_t(s,a)); the optimal baseline is b*=q_pi; u_t has a recursive form computable from offline data via FQE, enabling a doubly optimal estimator with unbiasedness under an enlarged policy space.

Key results

• On MuJoCo, DOpt needs 225 to 492 episodes versus 1000 for on-policy MC to reach the same accuracy, saving 50.8% to 77.5% of online samples.
• On Gridworld, the method reduces estimator variance by around 75% at both n^3=1,000 and n^3=27,000 states.

Measured effects reported in the paper

Surprising findings

• DR performs slightly better than ODI in the smaller Gridworld but slightly worse in the larger one, so no dominance relation holds between them.

Limitations

• Performance depends on offline data size; with very small offline data (e.g., a single tuple) the approximated behavior policy and baseline may be inaccurate.
• Analysis is limited to Monte Carlo estimators; extension to temporal difference learning is left to future work.

Experimental setup

Datasets: Gridworld (n^3 states, 1,000 and 27,000) MuJoCo tasks (Ant, Hopper, Inverted Double Pendulum, Inverted Pendulum, Walker)

Benchmarks: Gridworld MuJoCo

Metrics: relative estimation error relative variance episodes needed to match on-policy MC accuracy

Baselines: on-policy Monte Carlo offline data informed (ODI) estimator doubly robust (DR) estimator

Builds on: per-decision importance sampling Fitted Q-Evaluation doubly robust off-policy evaluation offline data informed policy evaluation

Closest papers by meaning

• Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning (NeurIPS 2022 · similarity 0.92)
• Efficient Multi-Policy Evaluation for Reinforcement Learning (AAAI 2025 · similarity 0.90)
• Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.90)
• Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning (ICLR 2025 · similarity 0.88)
• Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation (ICML 2025 · similarity 0.88)
• Beyond Expected Return: Accounting for Policy Reproducibility When Evaluating Reinforcement Learning Algorithms (AAAI 2024 · similarity 0.88)