Research map / Reinforcement learning / Offline RL
C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CIQL, an IQL-style safe offline RL method that avoids OOD by implicit cost updates, and C2IQL, which adds a cost reconstruction model for non-discounted costs and constraint-conditioned thresholds, evaluated on DSRL Bullet-Safety-Gym tasks.
Problem
Safe offline RL suffers from out-of-distribution value estimation, inaccurate discounted approximations of cumulative cost constraints, and fixed constraint thresholds that do not adapt to varying budgets.
Approach
Extends implicit Q-learning to constrained settings by deriving cost value updates under the same implicit policy (CIQL), then reconstructs non-discounted cumulative costs from multiple discounted cost values and conditions value and policy networks on the sampled constraint threshold (C2IQL).
Key results
- C2IQL achieves average normalized reward 0.74 and normalized cost 0.76 across Bullet-Safety-Gym tasks, versus CDT at 0.71 reward and 0.91 cost.
- Ablation on CarCircle shows removing the cost reconstruction model lowers reward and removing constraint conditioning yields more conservative costs; CIQL alone is overly conservative with very low reward.
- C2IQL is reported to remain safe across small, middle, and large thresholds where CDT fails to ensure safety at small thresholds in some environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use C2IQL instead of CDT | DSRL benchmarks · DSRL benchmarks (Avg) | normalized reward: 0.71 -> 0.74 | Helped |
| use C2IQL instead of CDT | DSRL benchmarks · DSRL benchmarks (Avg) | normalized cost: 0.91 -> 0.76 | Helped |
| add cost reconstruction to CIQL (ablation) | C2IQL · CarCircle | reward | Helped |
| add constraint conditioning to CIQL (ablation) | C2IQL · CarCircle | cost | Hurt |
| add cost reconstruction and constraint conditioning (ablation) | CIQL · CarCircle | reward | Helped |
| increase CRM input noise from 0.0 to 0.2 (ablation) | C2IQL · CarCircle | reward: 434.01 -> 442.08 | Helped |
| increase CRM input noise from 0.0 to 0.2 (ablation) | C2IQL · CarCircle | cost: 46.60 -> 48.30 | Hurt |
| increase CRM input noise from 1.0 to 4.0 (ablation) | C2IQL · CarCircle | reward: 421.82 -> 401.89 | Hurt |
| increase CRM input noise from 1.0 to 4.0 (ablation) | C2IQL · CarCircle | cost: 42.10 -> 36.20 | Helped |
| use a smaller value-function expectile parameter κ1 (ablation) | C2IQL · CarCircle | cost | Helped |
| use a larger policy-extraction expectile parameter κ2 (ablation) | C2IQL · CarCircle | reward | Helped |
Surprising findings
- CIQL without the reconstruction model or constraint conditioning converges to a safe but very low-reward policy with nearly zero cost.
Limitations
- Constraint thresholds are sampled from a predefined set rather than all thresholds being handled, to reduce computation.
- Cost reconstruction model accuracy affects performance; high noise degrades reward and cost behavior toward conservativeness.
Experimental setup
Datasets: DSRL Bullet-Safety-Gym datasets (Ant, Ball, Car, Drone; Run and Circle)
Benchmarks: Bullet-Safety-Gym (DSRL) SafetyGymnasium
Metrics: normalized reward return normalized cost return
Baselines: BCQ-Lag BEAR-Lag COptiDICE CPQ FISOR VOCE WSAC CDT
Builds on: Implicit Q-learning Implicit Diffusion Q-learning Constraints Penalized Q-learning DSRL
Closest papers by meaning
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model (NeurIPS 2024 · similarity 0.88)
- Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.