Research map / Reinforcement learning / Policy gradient and actor-critic

Learning Optimal Deterministic Policies with Stochastic Policy Gradients

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Trajectory reuse in policy gradients · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies global convergence of policy gradient with stochastic (action-based or parameter-based) white-noise exploration to the best deterministic policy, giving bounds linking exploration variance, sample complexity, and deployed deterministic performance.

Problem

Stochastic policies learned by policy gradient are deployed as deterministic policies, but theory lacks guarantees on the performance of the deterministic policy obtained when learning with noise-based exploration (action-based or parameter-based).

Approach

Formalizes white-noise-based action and parameter exploration, bounds the gap between the stochastic objectives and the deterministic objective, and proves last-iterate global convergence to the best deterministic policy under weak gradient domination, with the exploration variance tuned to trade sample complexity against deployed performance.

Key results

• Deployment gap bounds: |J_D - J_P| <= L_J sqrt(d_Theta) sigma_P and |J_D - J_A| <= L sqrt(d_A) sigma_A, with matching lower bound up to constants (0.28 factor).
• Global convergence of PGPE and GPOMDP to the best deterministic policy with sample complexity roughly O~(epsilon^-5) or O~(epsilon^-7) with variance tuned to epsilon, versus O~(epsilon^-3) for fixed variance to the best stochastic objective.
• Empirically on MuJoCo with linear policies, the best deterministic return depends on exploration variance: PGPE best at sigma_P^2 in {1,10}, GPOMDP best at sigma_A^2 in {1,10}; GPOMDP better on Hopper and HalfCheetah, PGPE better on Swimmer.

Measured effects reported in the paper

Surprising findings

• Learning the exploration variance jointly with policy parameters breaks the known sample complexity guarantees of GPOMDP, since smoothness grows inversely with variance.

Limitations

• Variance is fixed during learning; the theoretically optimal variance is not practical to set.
• Sample complexity bounds rely on weak gradient domination and smoothness assumptions and are polynomial but not optimal (epsilon^-5 to epsilon^-7 for deterministic objective).
• Experiments are limited to linear policies on three MuJoCo tasks.

Experimental setup

Datasets: MuJoCo Swimmer-v4 MuJoCo Hopper-v4 MuJoCo HalfCheetah-v4

Benchmarks: MuJoCo Swimmer-v4 MuJoCo Hopper-v4 MuJoCo HalfCheetah-v4

Metrics: deterministic policy return J_D gap between stochastic and deterministic objectives

Baselines: GPOMDP PGPE

Builds on: GPOMDP PGPE deterministic policy gradient Fisher non-degeneracy

Closest papers by meaning

• Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.94)
• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
• Expected Policy Gradients (AAAI 2018 · similarity 0.91)
• Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs (AAAI 2025 · similarity 0.89)
• On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Mollification Effects of Policy Gradient Methods (ICML 2024 · similarity 0.88)