Research map / Reinforcement learning / Offline RL
Q-Supervised Contrastive Representation: A State Decoupling Framework for Safe Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDQC, a safe offline RL framework that decouples states into reward- and cost-related representations learned via Q-supervised contrastive loss, and selects among three diffusion policies based on HJ-reachability safety assessment.
Problem
Safe offline RL policies violate safety constraints at test time due to out-of-distribution observations, since the combinations of reward-related and cost-related states are effectively infinite.
Approach
Learn reward- and cost-related state representations by contrastively clustering states whose optimal Q-values (across in-support actions) match, which coarsens the representation without model estimation (unlike bisimulation), and use decoupled representations with three diffusion policies chosen by HJ-based safety assessment.
Key results
- SDQC achieves near-zero normalized cost (e.g., 0.00 on CarGoal1, CarPush1) on many Safety-Gymnasium tasks, averaging 0.17 cost on SafetyGym tasks versus 1.25 for FISOR.
- On Bullet-Safety-Gym tasks, SDQC averages 0.08 normalized cost versus 0.10 for FISOR and 0.39 normalized reward.
- In generalization tests on Car agents, SDQC is reported as the only method that keeps cost from increasing when moving to unseen environments, with slight reward decay.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Q-supervised contrastive loss to critic and representation training (ablation) | SDQC · CarGoal2 | reward and cost | Helped |
| use an attention-based rather than MLP-based state encoder (ablation) | SDQC · CarGoal2 | reward and cost | Helped |
| set anchor number to 16 rather than 8 (ablation) | SDQC · CarGoal2 and CarPush2 | reward: 0.23 -> 0.22; 0.31 -> 0.28 | Mixed |
| set anchor number to 4 rather than 8 (ablation) | SDQC · CarGoal2 and CarPush2 | reward: 0.23 -> 0.20; 0.31 -> 0.25 | Hurt |
| set anchor number to 1 rather than 8 (ablation) | SDQC · CarGoal2 and CarPush2 | reward: 0.23 -> 0.15; 0.31 -> 0.10 | Hurt |
| remove contrastive learning by setting anchor number to 0 (ablation) | SDQC · CarGoal2 and CarPush2 | cost: 0.00 -> 0.86; 0.04 -> 2.15 | Hurt |
| increase contrastive temperature ν to 1 (ablation) | SDQC · PointGoal2 and CarPush2 | reward and cost: 0.31, 0.04 -> 0.22, 0.16; 0.29, 0.09 -> 0.20, 0.48 | Mixed |
| lower contrastive coefficient δ to 0.1 (ablation) | SDQC · PointGoal2 and CarPush2 | reward and cost: 0.31, 0.04 -> 0.31, 0.17; 0.29, 0.09 -> 0.30, 0.10 | Mixed |
| increase contrastive coefficient δ to 10 (ablation) | SDQC · PointGoal2 and CarPush2 | reward and cost: 0.31, 0.04 -> 0.23, 0.48; 0.29, 0.09 -> 0.22, 0.53 | Hurt |
| use only the reward policy πr instead of the three-policy combination (ablation) | SDQC · PointGoal1 and PointGoal2 | cost: 0.36 -> 4.92; 0.09 -> 13.28 | Hurt |
| use only the trade-off policy πto instead of the three-policy combination (ablation) | SDQC · PointGoal1 and PointGoal2 | reward: 0.35 -> 0.27; 0.29 -> 0.28 | Hurt |
| use only the cost policy πh instead of the three-policy combination (ablation) | SDQC · PointGoal1 and PointGoal2 | reward: 0.35 -> 0.01; 0.29 -> -0.11 | Hurt |
Surprising findings
- Naive reward-only policy and other single-policy ablations show that the combination of three policies is needed; some single policies achieve low cost but poor reward.
- Attention-based state encoder outperforms MLP encoder in clustering quality, and some hyperparameter settings (very small contrast temperature 0.01) cause training NaN/collapse.
Limitations
- Substantial computational cost from three training phases and complex attention-based encoders.
- Slower inference than non-autoregressive baselines.
- Hyperparameters (contrast coefficient, temperature, anchor number) are sensitive and tuned per domain.
Experimental setup
Datasets: DSRL benchmark
Benchmarks: DSRL Safety-Gymnasium Bullet-Safety-Gym
Metrics: normalized reward normalized cost
Baselines: BCQ-Lag CPQ COptiDICE CDT TREBI FISOR
Builds on: FISOR implicit Q-learning bisimulation weighted regressed diffusion models Safety Bellman operator (HJ reachability)
Closest papers by meaning
- Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.89)
- Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Feasibility Consistent Representation Learning for Safe Reinforcement Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.