Research map / Reinforcement learning / Offline RL
Boundary-to-Region Supervision for Offline Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Boundary-to-Region (B2R), which filters unsafe trajectories, realigns cost-to-go tokens to the fixed safety budget, and uses RoPE in a Decision Transformer for offline safe RL.
Problem
Sequence-model offline safe RL (e.g., Decision Transformer variants) symmetrically conditions on return-to-go and cost-to-go, leading to unreliable constraint satisfaction, especially on out-of-distribution cost trajectories and with sparse boundary-aligned data.
Approach
Treats CTG as a rigid boundary: filter to safe trajectories, then shift each trajectory's CTG sequence by a constant offset so every training sample starts at the fixed budget kappa, enabling region-wide supervision from all safe trajectories, with RoPE for relative temporal encoding.
Key results
- B2R satisfies safety constraints in 35 of 38 DSRL tasks and achieves the highest reward in 20 tasks.
- B2R has the best average reward across the DSRL suite in the main table (SafetyGym 0.42 / BulletGym 0.62 / MetaDrive 0.72) with lower average cost than CDT in the MetaDrive and BulletGym groups.
- Removing CTG realignment increases cost in nearly all environments while leaving reward unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| realign CTG tokens using uniform Shift rather than alternative redistribution strategies (ablation) | B2R · 9 representative DSRL tasks | reward and cost trade-off | Mixed |
| remove CTG realignment (ablation) | B2R · DSRL tasks including BallRun, CarCircle, and AntVelocity | cumulative cost | Hurt |
| remove CTG realignment (ablation) | B2R · DSRL tasks | reward | No clear effect |
| replace RoPE with standard absolute positional embeddings (ablation) | B2R · CarCircle, HopperVelocity, and easymean | reward | Hurt |
| replace RoPE with standard absolute positional embeddings (ablation) | B2R · CarCircle, HopperVelocity, and easymean | cost | Hurt |
| remove trajectory filtering (ablation) | B2R · CarCircle, HopperVelocity, and hardsparse | cost | Hurt |
| remove trajectory filtering (ablation) | B2R · DSRL tasks | reward | Mixed |
| use Scale instead of Shift for CTG realignment (ablation) | B2R · AntVelocity | reward | Helped |
| use Scale instead of Shift for CTG realignment (ablation) | B2R · HopperVelocity and sparse-cost environments | constraint violations | Hurt |
| use Avg instead of Shift for CTG realignment (ablation) | B2R · CarCircle and easymean | reward and cost performance | Hurt |
| use Rand instead of Shift for CTG realignment (ablation) | B2R · representative DSRL tasks | reward and cost performance | No clear effect |
| train on 5% rather than 100% of filtered safe trajectories (ablation) | B2R · BallCircle, cost threshold κ=10 | normalized cost: 0.59 -> 2.50 | Hurt |
Surprising findings
- Removing CTG realignment leaves reward identical to the full model, as expected since realignment only alters the cost signal.
- Simply buffering the CDT cost target (70-100% of kappa) does not reliably achieve safety, suggesting the gain is not merely a margin effect.
Limitations
- Performance degrades when safe trajectories are scarce, and the method relies on the availability of high-quality safe data.
- Fixed, non-adaptive uniform CTG realignment; theory relies on strong assumptions (bounded prediction error, safe-aligned data).
- Fails in very hard environments such as CarCircle1/2 and AntCircle, where most baselines also fail.
Experimental setup
Datasets: DSRL benchmark (Safety-Gymnasium, Bullet Safety-Gym, MetaDrive)
Benchmarks: DSRL
Metrics: normalized reward normalized cost
Baselines: BC-All BC-Safe CDT BCQ-Lag CPQ COptiDICE TraC FISOR OASIS LSPC FAWAC
Builds on: Decision Transformer Constrained Decision Transformer DSRL benchmark
Closest papers by meaning
- Safe Offline Reinforcement Learning using Behavior Regularisation and Latent Feasibility-Guidance (NeurIPS 2026 · similarity 0.90)
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.90)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.89)
- Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR 2024 · similarity 0.89)
- Feasibility Consistent Representation Learning for Safe Reinforcement Learning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.