Research map / Reinforcement learning / MDPs and planning

Offline Constrained Reinforcement Learning under Partial Data Coverage

NeurIPS 2026 · Reinforcement learning › MDPs and planning · approach: Offline RL sample complexity

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PDOCRL, an oracle-efficient primal-dual algorithm for offline constrained RL under partial coverage that optimizes policies directly via a decomposed LP and reparameterization, avoiding policy extraction from the unknown data distribution.

Problem

Offline constrained RL with general function approximation under partial data coverage, where intermediate policies may be unsupported and policy extraction from density ratios requires the unknown data distribution.

Approach

A Lagrangian decomposition introduces a copy of the occupancy measure and reparameterizes it as nu = pi times the Bellman-flow term, so the policy becomes an explicit optimization variable; under an all-policy value realizability and Slater condition, every restricted saddle point is optimal, and a primal-dual no-regret method solves the empirical saddle problem with standard oracles.

Key results

• PDOCRL average normalized reward 0.30 with normalized cost 0.30 across 8 BulletGym tasks, the only method with cost below 1 on all tasks (per paper claim).
• COptiDICE average reward 0.33 with cost 1.12; BC-Safe average reward 0.52 with cost 0.82 (safe on fewer tasks).
• Ablation with policy extraction: average reward -0.22 and cost 1.96 versus PDOCRL 0.30 and 0.30.

Surprising findings

• Saddle-point formulations with general function approximation can have spurious saddle points even when optimal policy, occupancy, and Q-functions are realizable.
• Empirical policy extraction from density ratios can select a suboptimal policy with constant probability even when the population extraction is optimal.

Limitations

• Requires all-policy value function realizability, which is strong and may not hold in practice.
• Requires known upper bound C* on concentrability and Slater margin phi.
• Empirical evaluation is limited to BulletGym, with high variance across seeds and mixed reward performance on several tasks.

Experimental setup

Datasets: DSRL BulletGym datasets

Benchmarks: BulletGym (DSRL) BallRun, CarRun, DroneRun, AntRun, BallCircle, CarCircle, DroneCircle, AntCircle

Metrics: normalized reward normalized cost

Baselines: BC-All BC-Safe CDT BCQ-Lag BEAR-Lag CPQ COptiDICE

Builds on: COptiDICE Lagrangian decomposition (Mehta & Meyn 2009) Gabbianelli et al. 2024 reparameterized decomposed Lagrangian PDCA (Hong et al. 2024) DSRL benchmark

Closest papers by meaning

• A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs (ICML 2024 · similarity 0.92)
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage (ICLR 2022 · similarity 0.91)
• Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Safe and Efficient: A Primal-Dual Method for Offline Convex CMDPs under Partial Data Coverage (NeurIPS 2024 · similarity 0.89)
• On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.89)
• POCE: Primal Policy Optimization with Conservative Estimation for Multi-constraint Offline Reinforcement Learning (CVPR 2024 · similarity 0.89)