Research map / Reinforcement learning / Offline RL
Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDT, a Decision Transformer conditioned on prefix and suffix STL robustness values (plus reward-to-go) to learn safe, high-reward offline policies that satisfy temporal logic specifications.
Problem
Offline safe RL with complex temporal and logical constraints is hard to capture with standard Markovian cost functions and cost-to-go conditioning.
Approach
Condition a Decision Transformer on STL robustness values of prefix (past trajectory) and suffix (future, target set in advance) tokens, so non-Markovian temporal specifications can be expressed and aligned with at evaluation time.
Key results
- SDT reaches average satisfaction rate 0.97 on Run, 0.86 on Circle, and 0.76 on Reach, above all baselines (best baseline RvS-R-rho 0.79, 0.80, 0.65).
- SDT achieves average Run reward 0.96 with cost 0.33, matching CDT's reward while improving satisfaction rate over CDT (0.92).
- Rescaling predicate weights (alpha_b, alpha_v up to 100) changes reward and satisfaction rate only marginally (e.g., rate 0.96-0.99).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition the policy on STL prefix and suffix robustness tokens rather than only return-to-go/cost conditions | SDT · Run-Average | Rate: 0.78$\pm$0.25 -> 0.97$\pm$0.04 | Helped |
| Condition the policy on STL prefix and suffix robustness tokens rather than only return-to-go/cost conditions | SDT · Circle-Average | Rate: 0.49$\pm$0.13 -> 0.86$\pm$0.1 | Helped |
| Condition the policy on STL prefix and suffix robustness tokens rather than only return-to-go/cost conditions | SDT · Reach-Average | Rate: 0.34 $\pm$ 0.15 -> 0.76 $\pm$ 0.19 | Helped |
| remove the prefix robustness token (ablation) | SDT · DSRL Run, Circle, and Reach | satisfaction rate | Hurt |
| remove the suffix robustness token (ablation) | SDT · DSRL Run, Circle, and Reach | satisfaction rate | Hurt |
| add a reward-prefix token to DT (ablation) | DT · D4RL Halfcheetah, Walker2D, and Hopper | normalized reward | Helped |
| use a linearly increasing target suffix instead of the fixed target suffix (ablation) | SDT · DSRL environments | satisfaction rate | Hurt |
| use the mean suffix from safe dataset trajectories as target (ablation) | SDT · DSRL environments | satisfaction rate | Hurt |
| use the maximum suffix from safe dataset trajectories as target (ablation) | SDT · DSRL environments | satisfaction rate | Hurt |
| rescale the boundary predicate by α_b=10 or α_b=100 (ablation) | SDT · Run | Reward: 0.96 $\pm$ 0.01 -> 0.95 $\pm$ 0.02; 0.96 $\pm$ 0.01 -> 0.93 $\pm$ 0.03 | No clear effect |
| rescale the velocity predicate by α_v=10 or α_v=100 (ablation) | SDT · Run | Rate: 0.97 $\pm$ 0.04 -> 0.97 $\pm$ 0.06; 0.97 $\pm$ 0.04 -> 0.99 $\pm$ 0.02 | No clear effect |
Surprising findings
- Lagrangian and Q-learning safe baselines often achieve higher rewards than conditioned baselines (stitching) but with frequent constraint violations.
- Adding an explicit reward prefix token to standard DT improved reward on D4RL medium and medium-replay datasets.
Limitations
- Reach environment lacks a relabeled cost, so only robustness-based baselines were compared there.
- Suffix target must be fixed in advance because robustness values lack an additive update, limiting autoregressive target specification.
- Evaluated only on Bullet-Safety-Gym and specifications constructed by the authors; the approach is limited to safety specifications over finite trajectories.
Experimental setup
Datasets: DSRL Bullet-Safety-Gym datasets (Ant-Run, Ball-Run, Drone-Run, Ball-Circle, Car-Circle, Drone-Circle, Reach variants) D4RL Gym (Halfcheetah, Walker2d, Hopper) for DT prefix experiment
Benchmarks: DSRL Bullet-Safety-Gym Run, Circle, Reach D4RL Gym (ablation of DT with reward prefix)
Metrics: normalized cumulative reward cumulative relabeled cost STL specification satisfaction rate
Baselines: CDT RvS-R RvS-RC BC-Safe BCQ-Lagrangian BEAR-Lagrangian CPQ COptiDICE
Builds on: Decision Transformer reward-conditioned supervised learning CDT OSRL
Closest papers by meaning
- DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL (ICLR 2025 · similarity 0.87)
- Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.87)
- When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · similarity 0.87)
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Regret-Free Reinforcement Learning for Temporal Logic Specifications (ICML 2025 · similarity 0.87)
- TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.