Research map / Reinforcement learning / Policy gradient and actor-critic

Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Constrained policy optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes C-PG, a regularized primal-dual policy-gradient framework for constrained RL with last-iterate convergence under weak gradient domination, plus action-based (C-PGAE) and parameter-based (C-PGPE) variants that handle risk-measure constraints.

Problem

Constrained reinforcement learning with policy-based primal-dual methods lacks last-iterate global convergence guarantees that are dimension-free, support multiple constraints, and cover continuous control and parameter-based exploration, and it does not handle risk-measure constraints.

Approach

Ridge-regularize the Lagrangian in the multipliers so the dual problem is strongly concave, then run alternating gradient descent-ascent under weak gradient domination; the analysis is exploration-agnostic and dimension-free, and a unified risk measure formulation lets the same algorithms enforce CVaR, mean-variance, and chance constraints in both AB and PB settings.

Key results

• C-PGAE reaches the constrained objective with fewer trajectories than sample-based NPG-PD and RPG-PD on the DGWW benchmark (5 runs).
• On CostLQR, C-PGAE and C-PGPE solve the constrained problem with fewer trajectories than NPG-PD2 and RPG-PD2, which need an additional inner critic loop of 500 trajectories.
• On Swimmer-v4 with risk constraints, C-PGPE learns a hyperpolicy with lower cost under CVaR than under the expected-cost constraint.

Measured effects reported in the paper

Surprising findings

• The mean-variance constraint attains the lowest cost on Swimmer-v4 but with poor performance (return reported in Table 3c).

Limitations

• Global convergence proof requires weak gradient domination assumption, which may not hold for risk-based objectives (acknowledged in Remark 4.1).
• Two-time-scale behavior when omega = O(epsilon) to get guarantees on the original problem; single time-scale rates are left open.
• Sample complexity lags lower bound of Vaswani et al. (2022).

Experimental setup

Datasets: Discrete Grid World with Walls (DGWW) CostLQR Swimmer-v4 Hopper-v4

Benchmarks: DGWW CostLQR Swimmer-v4

Metrics: average trajectory return average trajectory cost cost distribution sample complexity

Baselines: NPG-PD RPG-PD C-PGPE vs C-PGAE comparison

Builds on: NPG-PD RPG-PD PGPE GPOMDP unified risk measure (Bisi et al., 2022)

Closest papers by meaning

• Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.92)
• Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.91)
• Global Optimality for Constrained Exploration via Penalty Regularization (NeurIPS 2026 · similarity 0.91)
• Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.90)
• Deterministic Policies for Constrained Reinforcement Learning in Polynomial Time (NeurIPS 2024 · similarity 0.89)
• Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs (AAAI 2025 · similarity 0.89)