Research map / Reinforcement learning / Offline RL
Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WSAC, a two-player Stackelberg actor-critic for safe offline RL that uses importance-weighted Bellman error critics and an aggression-limited objective to guarantee safe robust policy improvement under single-policy l2 concentrability.
Problem
Safe offline RL with limited data coverage: existing methods need all-policy concentrability and cannot guarantee that the learned policy improves over a reference/behavior policy while staying safe.
Approach
Replace primal-dual dual-variable tuning with an aggression-limited penalized objective f_r - lambda*[f_c]_+ and adversarially trained (importance-weighted Bellman error) reward and cost critics, solved as a Stackelberg game with a no-regret actor, so only single-policy coverage of the reference policy is needed.
Key results
- On the average across four environments, WSAC reaches normalized reward 0.41 with normalized cost 0.51 (safe), while all baselines except BC-like ones have average cost above 1 or lower reward (e.g., BCQL 0.51 reward but cost 1.12).
- WSAC is the only method among those compared with cost below 1 on PointButton (0.67) and reports the best safe reward in BallCircle (0.75 with cost 0.27) and CarCircle (0.68, cost 0.59).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WSAC instead of Safe-BC | Offline safe RL · BallCircle | Reward: 0.61 -> 0.75 | Helped |
| Use WSAC instead of Safe-BC | Offline safe RL · CarCircle | Reward: 0.57 -> 0.68 | Helped |
| Use WSAC instead of Safe-BC | Offline safe RL · PointButton | Cost: 0.69 -> 0.67 | Helped |
| Use WSAC instead of BEARL | Offline safe RL · PointPush | Reward: 0.12 -> 0.07 | Hurt |
| Use WSAC instead of BEARL | Offline safe RL · PointPush | Cost: 0.90 -> 0.52 | Helped |
| Use WSAC instead of BCQL | Offline safe RL · Average | Reward: 0.51 -> 0.41 | Hurt |
| Use WSAC instead of Safe-BC | Offline safe RL · Average | Cost: 0.80 -> 0.51 | Helped |
Surprising findings
- Theory-driven baseline comparison shows that several existing safe offline RL methods (BCQL, BEARL, CPQ, COptiDICE) exceed the cost threshold in most environments, while higher-reward ones are unsafe.
Limitations
- Guarantees rely on function class realizability and boundedness assumptions (epsilon_1 = 0 for the clean result) and on a no-regret policy optimization oracle.
- Experiments are limited to four continuous control environments from one benchmark, with a single cost threshold in the main table.
Experimental setup
Datasets: offline safe RL benchmark (OSRL) datasets for BallCircle, CarCircle, PointButton, PointPush
Benchmarks: BallCircle CarCircle PointButton PointPush
Metrics: normalized reward normalized cost
Baselines: BC Safe-BC BCQL BEARL CPQ COptiDICE
Builds on: constrained actor-critic relative pessimism weighted Bellman error natural policy gradient / mirror descent as no-regret oracle
Closest papers by meaning
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.90)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation (ICML 2025 · similarity 0.89)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.89)
- Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.89)
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.