Research map / Reinforcement learning / Policy gradient and actor-critic

Global Optimality for Constrained Exploration via Penalty Regularization

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Constrained policy optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Policy Gradient Penalty (PGP), a single-loop policy-space quadratic-penalty method with pseudo-rewards for constrained maximum-entropy exploration, proving global last-iterate convergence under strong duality.

Problem

Maximizing state-action occupancy entropy under convex occupancy-measure constraints (safety, cost, imitation) with policy parameterization, which makes the problem non-convex and rules out Bellman-based methods; prior policy-gradient guarantees are only ergodic/weak-regret.

Approach

Quadratic-penalty reformulation of constrained entropy maximization in policy space; penalty gradients are rewritten as policy gradients w.r.t. pseudo-rewards so a single trajectory batch yields a stochastic gradient; hidden convexity and strong duality give last-iterate guarantees on objective and constraint.</�_idea>

Key results

• On FrozenLake, PGP satisfies Reg<=0.1 in 97.21% of final 1k epochs vs 65.33% for PDPG across 10 seeds.
• On SafeCartpole, PGP achieves swing-up >1 step in 87.23% of final 500 epochs vs 59.47% for PDPG (5 seeds).
• Theory: last-iterate guarantee with O~(eps^-6) trajectory rollouts for eps-optimal entropy and eps-feasibility under strong duality, with beta = O(1/eps).

Surprising findings

• Final entropy is essentially unchanged across a wide range of penalty parameters beta once feasibility is reached; larger beta mainly slows convergence.

Limitations

• Large theoretical sample complexity (O~(eps^-6)) and batch size O(eps^-4), possibly conservative.
• Theory covers finite MDPs and requires strong duality; continuous-control results are empirical only.
• Assumes smoothed entropy for analysis; step-size and batch-size selection in practice is not theoretically grounded.

Experimental setup

Datasets: FrozenLake grid world SafeCartpole PointMass

Benchmarks: FrozenLake SafeCartpole PointMass

Metrics: state-action occupancy entropy constraint violation feasibility fraction over final epochs swing-up fraction convergence epoch

Baselines: PDPG

Builds on: policy gradient theorem REINFORCE primal-dual policy gradient for convex RL (Ying et al. 2025) hidden convex optimization

Closest papers by meaning

• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
• Occupancy-based Policy Gradient: Estimation, Convergence, and Optimality (NeurIPS 2024 · similarity 0.90)
• Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.88)
• On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Parameter Exploration for RLVR via Variational Learning (NeurIPS 2026 · similarity 0.88)
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.88)