Research map / Reinforcement learning / Policy gradient and actor-critic
Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an unconstrained extended-MDP formulation that augments states with accumulated cost and applies reward penalties only to cost-violating trajectories, yielding Safe DQN and Safe SAC that handle expected-cost, VaR and CVaR constraints.
Problem
Constrained RL with trajectory-based cost constraints (expected cost, VaR, CVaR) is hard because existing methods convert them into local cost constraints that rely on error-prone cost estimates, yielding aggressive or conservative policies.
Approach
Augment the state with accumulated cost so far and apply reward penalties only to trajectories that exceed the cost budget; unlike Lagrangian methods that penalize all trajectories, increasing the penalty guarantees feasibility, and the same formulation covers expected, VaR and CVaR constraints.
Key results
- On the stochastic gridworld, Safe SAC and Safe DQN achieve similar reward to each other while satisfying the expected cost constraint after 1000 episodes, whereas Lyapunov violates it by more than twice the threshold.
- On Safety Gym navigation tasks, Safe SAC achieves the highest expected reward among methods satisfying cost constraints, while CPO matches its reward but violates the constraint.
- On the merge benchmark with CVaR constraints, Safe SAC marginally outperforms WCSAC in both expected reward and expected cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Safe SAC instead of WCSAC | Merge benchmark · Merge | expected reward | Helped |
| Use Safe SAC instead of WCSAC | Merge benchmark · Merge | expected cost | Helped |
| Use Safe SAC instead of other safe approaches | SafetyPointGoal1-v0 · SafetyPointGoal1-v0 | expected reward | Helped |
| Use Safe SAC instead of CPO | SafetyPointGoal1-v0 · SafetyPointGoal1-v0 | constraint satisfaction | Helped |
| Use Safe SAC instead of CUP or FOCOPS | SafetyPointGoal1-v0 · SafetyPointGoal1-v0 | expected reward | Helped |
| Use Safe SAC instead of other safe approaches | SafetyCarGoal1-v0 · SafetyCarGoal1-v0 | expected reward | Helped |
| Use Safe SAC instead of CPO | SafetyCarGoal1-v0 · SafetyCarGoal1-v0 | constraint satisfaction | Helped |
| Use Safe SAC instead of CUP or FOCOPS | SafetyCarGoal1-v0 · SafetyCarGoal1-v0 | expected reward | Helped |
| Use Safe SAC or Safe DQN instead of BVF | Gridworld · Stochastic Gridworld | expected reward | Helped |
| Use Safe SAC or Safe DQN instead of Lyapunov | Gridworld · Stochastic Gridworld | constraint satisfaction | Helped |
| Use Safe SAC instead of Safe DQN | Gridworld · Stochastic Gridworld | convergence speed | Helped |
| Use Safe SAC | Highway · Highway | expected reward | Helped |
Surprising findings
- The hard-constrained augmented-state baseline (Sootla et al.) is safe but too conservative, yielding lower reward than the reward-penalty approach in the toy example.
- CPO achieves reward on par with Safe SAC on Safety Gym but fails to satisfy the cost constraint.
Limitations
- Penalty parameter lambda must be tuned or dynamically adjusted, with heuristic schedule (0.95 decay with lower bound).
- Expected-cost results rely on cost estimates via Q-functions in Safe SAC action selection.
- VaR-CMDP not empirically evaluated because no existing baselines exist.
Experimental setup
Datasets: stochastic gridworld highway environment Safety Gym (SafetyPointGoal1, SafetyCarGoal1) GPIRL merge benchmark
Benchmarks: stochastic gridworld highway SafetyPointGoal1-v0 SafetyCarGoal1-v0 merge benchmark
Metrics: expected reward expected cost standard deviation over 5 runs
Baselines: Lyapunov BVF DQN FOCOPS CUP CPO WCSAC Lagrangian method Sootla et al. hard-constrained augmented state
Builds on: Soft Actor-Critic DQN Lagrangian constrained RL WCSAC
Closest papers by meaning
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.91)
- Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction (NeurIPS 2026 · similarity 0.90)
- Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.90)
- Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents (NeurIPS 2025 · similarity 0.90)
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.