Research map / Reinforcement learning / Visual and representation RL

From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning

NeurIPS 2024 · Reinforcement learning › Visual and representation RL · approach: Probabilistic and autonomous RL · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TTCT, which aligns trajectories with natural-language trajectory-level constraints via contrastive learning and decomposes episodic cost into per-step costs, so policies can be trained with predicted cost instead of hand-designed cost functions.

Problem

Safe RL with trajectory-level natural language constraints requires manually designed cost functions per constraint, and episodic (sparse) cost makes credit assignment hard.

Approach

Uses text as both constraint and training signal: a causal transformer trajectory encoder and a BERT text encoder are aligned contrastively (CLIP-style plus a within-trajectory loss) to predict violations, and a cost-assignment head decomposes the predicted episodic cost into dense per-step costs used for policy training with LoRA-tuned encoders.

Key results

• On Hazard-World-Grid with PPO_Lag, CP mode reduces Avg.C from 0.61 (GC) to 0.28 at similar reward (2.70 vs 2.71).
• Violation prediction on held-out pairs reaches accuracy 0.98 and AUC 0.98 for the text-trajectory alignment component.
• TTCT trained on Hazard-World-Grid transfers zero-shot to LavaWall with violation rates comparable to GC-trained policies.

Measured effects reported in the paper

Surprising findings

• Removing the cost assignment component causes large drops on SafetyGoal, e.g. PPO_Lag reward falls to -0.62 without CA.
• GC-trained CPPO_PID and FOCOPS violation rates rise during training on SafetyGoal.

Limitations

• Violation rate is not zero; method cannot fully eliminate violations.
• Contextual bottleneck: performance drops for very long trajectories due to transformer encoding limits.
• Experiments limited to simple grid and navigation environments with templated constraints.

Experimental setup

Datasets: Hazard-World-Grid trajectories from random policy SafetyGoal trajectories LavaWall (zero-shot)

Benchmarks: Hazard-World-Grid SafetyGoal LavaWall

Metrics: average episodic reward average episodic cost (violation rate) accuracy F1 AUC Pareto frontier

Baselines: PPO PPO_Lagrangian CPPO_PID FOCOPS

Builds on: CLIP PPO-Lagrangian CPPO-PID FOCOPS LoRA

Closest papers by meaning

• Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
• Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · similarity 0.88)
• Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.88)
• Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.88)
• Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.87)
• Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.87)