Research map / Reinforcement learning / Policy gradient and actor-critic

Action-Level Behavior Policy Optimization for Variance-Reduced Policy Gradients

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Variance-reduced policy gradients

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes optimizing the action-sampling distribution (a behavior policy distinct from the on-policy default) to reduce policy gradient variance, rather than only redesigning the estimator.

Problem

Policy gradient methods suffer high variance, limiting sample efficiency; existing variance reduction focuses on estimators under a fixed on-policy sampling distribution.

Approach

Treats the action-sampling distribution as a design variable and shows the on-policy default is not variance-optimal even without off-policy-ness constraints, then optimizes a behavior policy to reduce gradient variance.

Key results

• The abstract claims the on-policy default is not variance-optimal even without off-policy constraints; no numerical results are stated in the provided abstract.

Surprising findings

• The on-policy sampling default is not variance-optimal, even without off-policy constraints (as claimed in the abstract).

Experimental setup

Metrics: policy gradient variance

Builds on: policy gradient importance sampling

Closest papers by meaning

• Action-dependent Control Variates for Policy Optimization via Stein Identity (ICLR 2018 · similarity 0.91)
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines (ICLR 2018 · similarity 0.90)
• The Mirage of Action-Dependent Baselines in Reinforcement Learning (ICML 2018 · similarity 0.90)
• Efficient Multi-Policy Evaluation for Reinforcement Learning (AAAI 2025 · similarity 0.89)
• Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.88)
• Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.88)