Research map / Reinforcement learning / Policy gradient and actor-critic
Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PCRPO, a primal safe-RL method that soft-switches between reward and cost gradients via projection-based gradient manipulation with slack thresholds, and introduces a Safety-MuJoCo benchmark.
Problem
Safe RL must balance reward and safety optimization, but reward and cost gradients often conflict, causing oscillation between objectives and wasted samples in primal methods.
Approach
Applies gradient surgery-style projection to reward and cost gradients, averaging the projected gradients in a soft-switching rule governed by upper/lower slack values around the cost limit, with convergence/monotonic-improvement analysis; builds on CRPO's primal framework.
Key results
- On Omnisafe SafetyHopperVelocity-v1, PCRPO achieves higher reward than PCPO, CUP and PPOLag while maintaining safety, where CUP and PPOLag fail to satisfy the constraint.
- On SafetyAntVelocity-v1, PCRPO ensures complete safety with reward comparable to baselines and faster convergence.
- Ablations on slack settings show all PCRPO variants outperform CRPO in balancing reward and safety on SafetyWalker.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PCRPO instead of CRPO | Safety-MuJoCo tasks · SafetyWalker and SafetyHumanoidStandup | reward and cost performance | Helped |
| Use PCRPO instead of CRPO | Safety-MuJoCo tasks · SafetyReacher and SafetyHopper | reward and cost performance | Helped |
| Use PCRPO instead of PCPO, CUP, or PPOLag | Omnisafe · SafetyHopperVelocity-v1 | reward and safety | Helped |
| Use PCRPO instead of CUP or PPOLag | Omnisafe · SafetyAntVelocity-v1 | safety | Helped |
| Use PCRPO instead of PCPO | Omnisafe · SafetyAntVelocity-v1 | reward | Helped |
| Use PCRPO instead of the baselines | Omnisafe · SafetyHopperVelocity-v1 and SafetyAntVelocity-v1 | convergence speed | Helped |
| Use PCRPO-2SR slack settings instead of CRPO (ablation) | PCRPO · SafetyWalker | reward and safety balance | Helped |
| Use PCRPO-3SR-G slack settings instead of CRPO (ablation) | PCRPO · SafetyWalker | reward and safety balance | Helped |
| Use PCRPO-4S-F static slack settings instead of CRPO (ablation) | PCRPO · SafetyWalker | cost | Helped |
| Use PCRPO-4S-G dynamic slack settings instead of CRPO (ablation) | PCRPO · SafetyWalker | reward | Helped |
| Use PCRPO gradient manipulation instead of SCRPO gradient surgery (ablation) | PCRPO · SafetyHumanoidStandup | reward and safety performance | Helped |
Surprising findings
- Projection-based PCRPO outperforms SCRPO, which uses the PCGrad-style gradient surgery, suggesting that handling non-conflicting gradients also matters.
Limitations
- Theoretical guarantees are stated for specific angle ranges and under Lipschitz assumptions, with other cases only sketched.
- Stochastic gradient manipulation may not converge; the correlation-reduction fix is described only loosely.
- Results rely on hyperparameters fine-tuned by others and a single modest CPU/GPU setup, with limited task coverage.
Experimental setup
Datasets: Safety-MuJoCo Omnisafe (SafetyHopperVelocity-v1, SafetyAntVelocity-v1)
Benchmarks: Safety-MuJoCo Benchmark Omnisafe
Metrics: episodic reward episodic cost
Baselines: CRPO PCPO CUP PPO Lagrangian PDO SCRPO
Builds on: CRPO gradient surgery (PCGrad) TRPO PPO
Closest papers by meaning
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.91)
- Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.89)
- Risk-Sensitive Reward-Free Reinforcement Learning with CVaR (ICML 2024 · similarity 0.89)
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.89)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.