Research map / Reinforcement learning / Offline RL
Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAPS, a wrapper that trains a shared-representation set of reward/cost trade-off policies around offline RL (IQL or SAC+BC) and switches among them at test time to satisfy any cost threshold without retraining.
Problem
Offline safe RL where the cost threshold may change at deployment, while most prior OSRL methods fix the threshold at training time and must be retrained for each threshold.
Approach
Train multiple policies (reward-only, cost-only, and linear reward-cost scalarizations) with a shared actor body on reduced offline RL runs, then at each state filter policy actions by a cost Q-function against the remaining budget and pick the highest-reward safe action; fall back to the least-cost policy if none is safe.
Key results
- CAPS(IQL) meets the cost constraint on 34 of 38 tasks, versus 19 of 38 for the second-best method CDT, and achieves the highest reward among safe agents on 18 tasks.
- CAPS(SAC+BC) is safe on 30 of 38 tasks under the main cost threshold setting.
- Across cost-limit configurations, CAPS(IQL) is safe on 18 to 33 of 38 tasks, outperforming CDT (11 to 25 of 38).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use shared policy representation instead of independently trained agents (ablation) | CAPS(SAC+BC) · CarGoal1 | normalized reward: 0.00 to 0.28 | Helped |
| use shared policy representation instead of independently trained agents (ablation) | CAPS(SAC+BC) · AntVelocity | normalized reward: -0.78 to 0.89 | Helped |
| use shared policy representation instead of independently trained agents (ablation) | CAPS(IQL) · CarGoal2 | normalized reward: 0.15 to 0.16 | Helped |
| increase number of policies from 2 to 4 (ablation) | CAPS(IQL) · 38 DSRL tasks | normalized reward | Helped |
| increase number of policies from 4 to 8 (ablation) | CAPS(IQL) · 38 DSRL tasks | normalized reward | No clear effect |
| increase the policy count from 2 to 8 heads (ablation) | CAPS(IQL) · PointButton2 | normalized cost: 0.75 to 1.01 | Hurt |
| use reward FQE for reward estimation while retaining offline-RL cost Q (ablation) | CAPS · 38 DSRL tasks | performance | No clear effect |
| use reward-cost FQE estimates for both policies (ablation) | CAPS · 38 DSRL tasks | normalized reward | Hurt |
| increase cost limits from {5, 10} to {10, 20} (ablation) | CAPS(IQL) · 38 DSRL tasks | tasks satisfying cost constraint: 18 / 38 to 28 / 38 | Helped |
Surprising findings
- Sharing the actor backbone across heads substantially helps SAC+BC, where separate agents gave poor rewards (e.g., AntVelocity reward -0.78 separate vs 0.89 shared).
- Off-policy evaluation (FQE) for cost Q-functions yielded overly conservative policies, so the paper uses the offline RL cost Q-function for filtering.
Limitations
- Safety guarantee relies on perfect cost Q-function and an assumed bounded optimal-cost variation epsilon.
- Performance at low cost thresholds (e.g., {5,10}) is modest, with CAPS safe on only 15 to 18 of 38 tasks.
- Using more than two heads adds hyperparameters and sometimes produces unsafe agents.
Experimental setup
Datasets: DSRL benchmark (Safety-Gymnasium, Bullet-Safety-Gym, MetaDrive)
Benchmarks: DSRL 38 tasks (Safety-Gymnasium, Bullet-Safety-Gym, MetaDrive)
Metrics: normalized reward normalized cost number of safe tasks
Baselines: BC BEAR-Lag CPQ CDT COptiDICE
Builds on: Implicit Q-Learning SAC+BC DSRL Constrained Decision Transformer
Closest papers by meaning
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.93)
- Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR 2024 · similarity 0.91)
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.91)
- An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.