Research map / Reinforcement learning / Offline RL

POCE: Primal Policy Optimization with Conservative Estimation for Multi-constraint Offline Reinforcement Learning

CVPR 2024 · Reinforcement learning › Offline RL · approach: Pessimism and uncertainty in offline RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes POCE, a primal policy optimization method with conservative Q-estimation (including a conditional Bellman operator) for offline RL satisfying both cumulative and state-wise cost constraints.

Problem

Offline safe RL where policies must satisfy cumulative and state-wise cost constraints simultaneously, without environment interaction; existing constrained offline RL methods handle only cumulative costs and Lagrangian methods are unstable and costly to tune, and OOD actions cause extrapolation err

Approach

Reformulates state-wise costs as cost increments over an augmented state (maximum state-wise cost), optimizes constraints via primal policy gradient switching between reward maximization and cost minimization (avoiding Lagrange multipliers), and uses a conditional Bellman operator that treats in-distribution and OOD actions differently to estimate cost Q-values without underestimation.

Key results

• On PointGoal with safety factor 0.8, POCE reaches reward 12.74 with cost 4.37 and cost rate 0.09%, versus BCQ-Lag reward 11.27, cost 6.05, cost rate 0.20%.
• Averaged over PointGoal and CarGoal across safety factors, POCE attains mean reward 13.06 and mean cost 4.51 (cost rate 0.09%), outperforming BCQ-Lag, CCRR, and CPQ-Lag on safety.
• Ablation: removing conservative cost-Q estimation (POCE-Qx) breaks the cumulative cost constraint and raises state-wise violation rate in PointGoal.

Measured effects reported in the paper

Surprising findings

• Removing conservative reward Q-estimation drastically lowers reward, while removing conservative cost Q-estimation slightly raises reward but violates constraints.

Limitations

• No standardized multi-constraint offline RL benchmark; the authors construct their own cost scaling and datasets.
• Baselines are self-constructed multi-constraint variants of existing single-constraint methods.
• Results are reported over only three random seeds and three tasks.

Experimental setup

Datasets: Safety-gym PointGoal Safety-gym CarGoal MuJoCo AntVelocity offline datasets collected with safety-factor behavior policies

Benchmarks: PointGoal CarGoal AntVelocity

Metrics: reward cumulative cost cost rate (state-wise violation rate)

Baselines: BCQ-Lag CCRR CPQ-Lag

Builds on: CQL MCQ CRPO SCPO BCQ

Closest papers by meaning

• Offline Constrained Multi-Objective Reinforcement Learning via Pessimistic Dual Value Iteration (NeurIPS 2021 · similarity 0.90)
• Off-Policy Primal-Dual Safe Reinforcement Learning (ICLR 2024 · similarity 0.90)
• A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs (ICML 2024 · similarity 0.89)
• Model-based Reinforcement Learning for Confounded POMDPs (ICML 2024 · similarity 0.89)
• Offline Constrained Reinforcement Learning under Partial Data Coverage (NeurIPS 2026 · similarity 0.89)
• OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.88)