Research map / Reinforcement learning / Offline RL
Offline Safe Reinforcement Learning Using Trajectory Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TraC, which partitions an offline safe-RL dataset into desirable and undesirable trajectories and trains a policy with a sigmoid-based contrastive trajectory classification loss, avoiding min-max optimization.
Problem
Offline safe RL methods rely on per-step cost constraints derived from global trajectory constraints, leading to overly conservative or unsafe policies, and often use unstable min-max optimization.
Approach
Label trajectories as desirable (safe, high-return) or undesirable (unsafe or low-return safe) and learn a policy by maximizing a regret-based, KTO-style sigmoid classification likelihood of trajectory scores (log-ratio to a BC reference policy), with no min-max optimization.
Key results
- TraC averages normalized reward 0.40 and cost 0.92 on SafetyGym, satisfying the safety constraint where baselines like CDT (cost 1.06) do not.
- TraC is safe on all BulletGym and MetaDrive tasks in the averaged results, with BulletGym reward 0.61 and cost 0.52.
- Removing BC reference-policy pretraining drops BulletGym average reward from 0.61 to 0.12 and raises cost from 0.52 to 2.71.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TraC instead of the strongest-reward baseline CDT | offline safe RL · SafetyGym | normalized reward: 0.54 -> 0.4 | Hurt |
| use TraC instead of BC-Safe | offline safe RL · SafetyGym | normalized cost: 0.75 -> 0.92 | Hurt |
| use TraC instead of BCQ-Lag | offline safe RL · BulletGym | normalized reward: 0.74 -> 0.61 | Hurt |
| use TraC instead of BC-Safe | offline safe RL · BulletGym | normalized cost: 0.82 -> 0.52 | Helped |
| use TraC instead of COptiDICE | offline safe RL · MetaDrive | normalized reward: 0.58 -> 0.53 | Hurt |
| use TraC instead of CPQ | offline safe RL · MetaDrive | normalized cost: 0.06 -> 0.55 | Hurt |
| vary δ from 0 to 1.0 (ablation) | TraC · AntCircle | normalized reward: 0.46, 0.47, 0.47, 0.46 | No clear effect |
| vary η from 0.25 to 2.0 (ablation) | TraC · AntCircle | normalized reward: 0.47, 0.47, 0.46, 0.46 | Hurt |
| remove π_ref pretraining (ablation) | TraC · BulletGym average | normalized reward: 0.61 -> 0.12 | Hurt |
| remove π_ref pretraining (ablation) | TraC · BulletGym average | normalized cost: 0.52 -> 2.71 | Hurt |
| train with only undesirable trajectories instead of both subsets (ablation) | TraC · AntCircle, AntRun, BallCircle | normalized reward | Hurt |
| train with only desirable trajectories instead of both subsets (ablation) | TraC · DSRL tasks | overall performance | Hurt |
Surprising findings
- CDT's highest average reward on MetaDrive comes from two unsafe tasks; the average reward drops to 0.29 when only safe policies are considered.
- Training only on undesirable trajectories yields very low rewards and sometimes very high costs.
Limitations
- Performance depends on dataset partition hyperparameters x%, y%, and eta, which are set per task empirically.
- Reported safety is not met on all SafetyGym tasks.
- Policies are Gaussian with fixed variance, so the log-likelihood reduces to a squared-error term.
Experimental setup
Datasets: DSRL benchmark datasets
Benchmarks: DSRL SafetyGymnasium DSRL BulletSafetyGym DSRL MetaDrive
Metrics: normalized reward normalized cost
Baselines: BC-All BC-Safe CDT BCQ-Lag BEAR-Lag CPQ COptiDICE
Builds on: CPL KTO CDT DSRL
Closest papers by meaning
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model (ICLR 2024 · similarity 0.91)
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.91)
- FOSP: Fine-tuning Offline Safe Policy through World Models (ICLR 2025 · similarity 0.90)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.90)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.