Research map / Reinforcement learning / Policy gradient and actor-critic

Feasibility Consistent Representation Learning for Safe Reinforcement Learning

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Safety layers and shielding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FCSRL, a representation learning framework for safe RL that combines a SimSiam-style transition dynamics loss with a distributional feasibility-score prediction loss, plugged into PPO-Lagrangian and TD3-Lagrangian.

Problem

Cost estimation in safe RL is hard because sparse constraint signals make safety-relevant state representations and cost value estimates noisy and poorly learned, hurting the reward-safety trade-off.

Approach

Learn a state embedding with a dynamics consistency loss (target encoder, SimSiam-style) plus an auxiliary feasibility-score prediction (max discounted cost, a smoother target than cost value) trained with discrete distributional regression, then feed the target embedding to the safe RL policy and critics.

Key results

• On vector-state tasks, FCSRL achieves reward 24.4±1.4 at cost 9.3±5.8 on PointGoal1 and 27.6±2.0 on CarGoal1 (cost limit 10), exceeding CPO, PPO-Lag, FOCOPS and CVPO.
• On image-based tasks, FCSRL outperforms CURL, SPR and value consistent model with TD3-Lag, where CURL nearly fails on CarGoal1.
• Feasibility-score embeddings give lower prediction MSE (0.139 vs 0.217) than the value consistent model's embeddings.

Measured effects reported in the paper

Surprising findings

• Value-consistency objective on cost value gave limited benefit in the safe RL setting, while the feasibility target was easier to predict from the learned embedding.
• Feeding raw state together with the learned embedding to the policy was needed; using only the embedding caused a large reward drop.

Limitations

• Feasibility score is used only as representation auxiliary signal, not for state-wise safety in the policy itself (stated as future work).
• Reward gains are small or within standard deviation on several tasks (e.g., Goal1 where margins are small), and the ablation shows overlapping error bars.
• Relies on binary cost and experiments limited to Safety-Gymnasium tasks.

Experimental setup

Datasets: Safety-Gymnasium PointGoal1 PointButton1 PointPush1 PointGoal2 CarGoal1 CarButton1 Goal1 Push1

Benchmarks: Safety-Gymnasium

Metrics: normalized reward normalized cost MSE of cost value and feasibility prediction

Baselines: Raw state input Forward raw model Forward latent model Inverse model Temporal contrastive learning SALE Value consistent model CURL SPR CPO PPO-Lag FOCOPS

Builds on: SALE SPR CURL PPO-Lagrangian TD3-Lagrangian SimSiam

Closest papers by meaning

• Safety Representations for Safer Policy Learning (ICLR 2025 · similarity 0.91)
• Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.90)
• Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
• A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.89)
• Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.89)