Research map / Reinforcement learning / Policy gradient and actor-critic
Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PES, a phase-based policy gradient method with a deterministic decreasing stochasticity schedule, proving last-iterate convergence to the optimal deterministic policy at O~(eps^-5), and analyzes jointly learned stochasticity (SL-PG) converging to optimal stochastic policies at O~(eps^-3).
Problem
Policy gradient methods with fixed stochasticity guarantee convergence to deterministic policies only when the noise is set to the target suboptimality from the start, whereas practice uses dynamically changing stochasticity with no theoretical guarantees.
Approach
Reduce exploration noise via a deterministic phase schedule sigma_p = sigma_max (p+1)^(-y), running a fixed-stochasticity PG subroutine per phase so each phase starts from a near-optimal parameter; separately analyze jointly learning the stochasticity via a reparameterization (SL-PG).
Key results
- PES achieves last-iterate convergence to the optimal deterministic policy with sample complexity O~(eps^-5) under gradient domination assumptions.
- SL-PG converges to the optimal stochastic (hyper)policy with O~(eps^-3 sigma_min^-2) sample complexity under stronger assumptions.
- On Swimmer-v5, PES-B achieves the best PB convergence; under AB, PES can get stuck in local optima while SL-PG adapts sigma and avoids it (Table 1: AB deterministic deployment 49.83 for SL-PG vs 28.93 for PES-A).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reduce stochasticity with a phased deterministic schedule | PG methods (PES) · optimal deterministic policy | sample complexity: r̃O(ϵ⁻⁵) | Helped |
| jointly learn policy parameters and stochasticity | PG methods (SL-PG) · optimal stochastic (hyper)policy | sample complexity: r̃O(ϵ⁻³σ_min⁻²) | Helped |
| use schedule B with 1 iteration per phase instead of schedule A with 200 iterations per phase (ablation) | PES · Swimmer-v5 deterministic deployment (PB) | return: 61.14 ± 1.1 -> 60.40 ± 2.31 | Hurt |
| use schedule B with 1 iteration per phase instead of schedule A with 200 iterations per phase (ablation) | PES · Swimmer-v5 deterministic deployment (AB) | return: 28.93 ± 9.89 -> 29.24 ± 8.74 | No clear effect |
| use phased schedule A instead of jointly learning stochasticity | PG methods · Swimmer-v5 deterministic deployment (PB) | return: 61.43 ± 2.73 -> 61.14 ± 1.1 | No clear effect |
| use phased schedule A instead of jointly learning stochasticity | PG methods · Swimmer-v5 deterministic deployment (AB) | return: 49.83 ± 13.38 -> 28.93 ± 9.89 | Hurt |
| use schedule B with 1 iteration per phase instead of schedule A with 200 iterations per phase (ablation) | PES · Swimmer-v5 | convergence performance (JP) | Helped |
| use schedule B with 1 iteration per phase instead of schedule A with 200 iterations per phase (ablation) | PES · Swimmer-v5 | convergence performance (JA) | Hurt |
Surprising findings
- Under action-based exploration, PES can get stuck in local optima when the per-phase iterations are insufficient, whereas SL-PG avoids this.
Limitations
- SL-PG convergence requires stronger assumptions and only reaches optimal stochastic policies unless learned sigma is bounded by eps.
- PES may get stuck in local optima if phase lengths or schedule exponent are poorly chosen.
- Experiments are limited to a single MuJoCo environment (Swimmer-v5).
Experimental setup
Datasets: Swimmer-v5
Benchmarks: Swimmer-v5
Metrics: expected return J deterministic deployment performance stochasticity sigma over iterations
Baselines: GPOMDP PGPE SL-PG
Builds on: GPOMDP PGPE Montenegro et al. 2024 white noise PG framework HARPG
Closest papers by meaning
- Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.94)
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policies (ICML 2023 · similarity 0.91)
- Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs (AAAI 2025 · similarity 0.89)
- On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Accelerated Policy Gradient: On the Convergence Rates of the Nesterov Momentum for Reinforcement Learning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.