Research map / Reinforcement learning / Policy gradient and actor-critic

Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Safety layers and shielding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ASCPO, a trust-region policy search method that bounds the mean plus k-times standard deviation of the maximum state-wise cost to guarantee high-probability state-wise constraint satisfaction without dynamics assumptions, evaluated on Safety Gym-style GUARD robot tasks.

Problem

Enforcing hard state-wise safety constraints in model-free RL is hard: expectation-based state-wise methods allow rare severe violations, while guaranteed approaches need strong dynamics assumptions.

Approach

Bounds the upper probability of the maximum state-wise cost by E[D] + kV[D] (mean plus k standard deviations), derives surrogate bounds for mean and variance under MMDP, and optimizes these inside a TRPO-style trust region so the constraint holds with high probability without knowledge of dynamics.

Key results

• ASCPO achieves near-zero episodic cost (e.g., Point-8-Hazard cost 0.0413 vs 0.0919 for SCPO and 0.1961 for CPO) with comparable reward (2.4785 vs TRPO 2.4603).
• Across 19 test suites, ASCPO attains the highest synthesised score relative to TRPO among the 12 compared algorithms (Figure 8).
• PASCPO, a PPO-based variant, outperforms fine-tuned PPO-Lagrangian in four representative tasks with similar resource usage.

Surprising findings

• Lagrangian-based baselines showed low cost rates mainly because episodes lengthened during poor early learning, masking poor reward and cost convergence.
• Monotonic-descent loss helps on MMDP-based methods but has no visible effect on non-MMDP CPO.

Limitations

• Practical implementation treats several theoretically derived terms (such as infinity norms and maximum KL) as hyperparameters, so guarantees are approximate.
• Each metric averaged over only two seeds in tables, and evaluation limited to simulated MuJoCo/Safety Gym tasks.

Experimental setup

Datasets: GUARD benchmark (Safety Gym-based robot tasks)

Benchmarks: GUARD 19 test suites (Point, Swimmer, Arm3, Drone, Humanoid, Ant, Walker with Hazard, Pillar, Ghost, 3DHazard constraint

Metrics: average episode return average episodic cost cost rate synthesised score

Baselines: TRPO CPO SCPO TRPO-Lagrangian TRPO-FAC TRPO-IPO PCPO TRPO-SL TRPO-USL TRPO-CVaR CPO-CVaR PPO-Lagrangian

Builds on: TRPO CPO SCPO MMDP Safety Gym

Closest papers by meaning

• Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.90)
• Near-Optimal Sample Complexity for Online Constrained MDPs (NeurIPS 2025 · similarity 0.89)
• Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · similarity 0.89)
• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
• Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.89)
• Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.89)