Research map / Reinforcement learning / Policy gradient and actor-critic
Off-Policy Primal-Dual Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAL, an off-policy primal-dual safe RL method combining UCB-based conservative cost estimation (bootstrapped Q-ensemble) with an augmented Lagrangian local policy convexification, evaluated on Safety-Gym, velocity-constrained MuJoCo, and an oCPM advertising task.
Problem
Primal-dual safe RL with off-policy learning suffers from cost underestimation caused by value estimation error, leading to constraint violations and unstable Lagrange multiplier updates, and poor sample efficiency of on-policy methods.
Approach
Replace the expected cost in the Lagrangian with an upper confidence bound from a bootstrapped cost-critic ensemble (conservative policy optimization), and apply an augmented Lagrangian term that rectifies the multiplier in the policy gradient (local policy convexification), so uncertainty shrinks gradually and the conservative boundary approaches the true constraint boundary.
Key results
- On benchmark tasks CAL matches state-of-the-art on-policy methods' asymptotic reward with far fewer samples and significantly fewer constraint violations during training.
- In the oCPM semi-batch experiment, CAL reached 656.31k payment in Week 1 vs 433.33k for the best baseline CUP, with lower ROI violation in Week 3 (18.15% vs 20.88%).
- Ablation shows k=0 (no conservatism) leads to significantly more constraint violations and failure to reach a constraint-satisfying policy on PointButton2 and CarButton1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAL instead of the strongest listed baseline | oCPM auto-bidding · oCPM, Weeks 1–4 | total advertiser payments | Helped |
| Use CAL instead of the strongest listed baseline | oCPM auto-bidding · oCPM, Weeks 3–4 | ROI violation rate | Helped |
| Use CAL instead of the strongest listed baseline | oCPM auto-bidding · oCPM, Weeks 1–2 | ROI violation rate | Hurt |
| Add cost conservatism by setting k above 0 (ablation) | CAL · PointButton2 and CarButton1 | constraint satisfaction | Helped |
| Increase k; excessive conservatism can hinder reward maximization (ablation) | CAL · tested benchmark tasks | reward | Mixed |
| Add the ALM modification by setting c above 0 (ablation) | CAL · tested benchmark tasks | performance stability | Helped |
| Increase c (ablation) | CAL · tested benchmark tasks | performance oscillations | Helped |
| Increase c (ablation) | CAL · tested benchmark tasks | reward improvement | Hurt |
| Set c excessively large (ablation) | CAL · tested benchmark tasks | reward improvement | Hurt |
| Increase ensemble size E (ablation) | CAL · Hopper, PointButton2, and CarButton2 | cost | Helped |
| Increase ensemble size E (ablation) | CAL · Hopper and PointButton2 | overall performance | No clear effect |
Surprising findings
- Increasing UTD ratio for baselines improves reward but does not improve their constraint satisfaction, whereas CAL benefits from large UTD.
- Naive SAC-Lag shows severe cost underestimation, causing poor constraint satisfaction.
Limitations
- Ensemble of cost critics adds computational burden; larger ensembles trade off with compute.
- Hyperparameters k and c require task-dependent tuning (e.g., k=0.8 for PointPush1, c=100/1000 for some MuJoCo tasks).
- Evaluated only in the online and semi-batch settings; fully offline setting left for future work.
Experimental setup
Datasets: Safety-Gym velocity-constrained MuJoCo oCPM advertising bidding records (204.8M, 47244 advertisers)
Benchmarks: Safety-Gym velocity-constrained MuJoCo oCPM auto-bidding
Metrics: episodic reward episodic cost / constraint violation total advertiser payments ROI violation rate
Baselines: SAC-PIDLag WCSAC CVPO PDO RCPO IPO CRPO CUP
Builds on: SAC SAC-Lag augmented Lagrangian method
Closest papers by meaning
- POCE: Primal Policy Optimization with Conservative Estimation for Multi-constraint Offline Reinforcement Learning (CVPR 2024 · similarity 0.90)
- Safe and Efficient: A Primal-Dual Method for Offline Convex CMDPs under Partial Data Coverage (NeurIPS 2024 · similarity 0.90)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.89)
- Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation (ICML 2024 · similarity 0.89)
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.