Research map / Reinforcement learning / Policy gradient and actor-critic
Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ESPO, a primal safe-RL method that switches between reward, cost, and trade-off updates based on reward-cost gradient conflict and adapts per-iteration sample size accordingly, with convergence and sample-complexity analysis in the tabular softmax setting.
Problem
Safe RL is sample-inefficient because primal methods use a fixed sample size across all iterations, wasting samples in simple regions and under-exploring in conflicting-gradient regions.
Approach
Uses reward-cost gradient conflict (angle > 90 degrees) as a signal to increase the sample size in conflict and decrease it under alignment, within a three-mode PCRPO-style primal update that depends on the cost value relative to a soft constraint region.
Key results
- ESPO reaches 5.7M samples on SafetyReacher-v4 vs 8M for CRPO and PCRPO, about 29% fewer.
- On Omnisafe SafetyHopperVelocity-v1, ESPO uses 7.3M samples vs 10M for PCPO, CUP, and PPOLag, about 27% fewer.
- Theory gives convergence rate O~(sqrt(SA/((1-gamma)^3 T))) matching CRPO, and ESPO spends 99.4% of steps optimizing reward inside the safe region vs 35.6% for CRPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use gradient-conflict-based adaptive sample manipulation | ESPO · SafetyReacher-v4 | sampling steps: 5.7 M vs 8 M | Helped |
| use gradient-conflict-based adaptive sample manipulation | ESPO · SafetyWalker-v4 | sampling steps: 6.2 M vs 8 M | Helped |
| use gradient-conflict-based adaptive sample manipulation | ESPO · SafetyHumanoidStandup-v4 | sampling steps: 5.1 M vs 8 M | Helped |
| use gradient-conflict-based adaptive sample manipulation | ESPO · SafetyHopperVelocity-v1 | sampling steps: 7.3 M vs 10 M | Helped |
| use gradient-conflict-based adaptive sample manipulation | ESPO · SafetyAntVelocity-v1 | sampling steps: 7.6 M vs 10 M | Helped |
| use ESPO instead of primal baselines | Safety-MuJoCo tasks · SafetyReacher-v4 and SafetyWalker-v4 | reward | Helped |
| use ESPO instead of primal-dual baselines | OmniSafe tasks · SafetyHopperVelocity-v1 and SafetyAntVelocity-v1 | reward | Helped |
| use ESPO instead of baselines that struggle to maintain safety | OmniSafe · SafetyHopperVelocity-v1 | safety | Helped |
Surprising findings
- Reducing samples does not always proportionally reduce training time, since this depends on benchmark and parallelism settings.
Limitations
- Theoretical analysis is in the tabular softmax setting, not the deep function approximation used in experiments.
- Sample reduction does not translate linearly into training time reduction.
- Plans to test in real-world control applications remain future work.
Experimental setup
Datasets: Safety-MuJoCo Omnisafe
Benchmarks: Safety-MuJoCo Omnisafe
Metrics: reward cost constraint satisfaction number of samples training time
Baselines: CRPO PCRPO PCPO CUP PPOLag
Builds on: CRPO PCRPO PPO-Lagrangian natural policy gradient
Closest papers by meaning
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.92)
- Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation (AAAI 2024 · similarity 0.91)
- Feasible Policy Optimization for Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning (ICLR 2025 · similarity 0.90)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.