Research map / Reinforcement learning / Policy gradient and actor-critic
Constrained Reinforcement Learning Under Model Mismatch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Robust Constrained Policy Optimization (RCPO), a primal-only trust-region method for constrained RL under transition-kernel uncertainty that guarantees worst-case reward improvement and constraint satisfaction per iteration.
Problem
Constrained RL policies that satisfy a utility constraint in the training environment may violate it in a real environment due to model mismatch; prior robust constrained methods lack monotonic guarantees or are limited to tabular settings.
Approach
Generalizes the performance difference lemma and trust-region CPO to robust MDPs with (s,a)-rectangular KL-type uncertainty sets, approximating worst-case advantage and visitation with the current policy's worst-case kernel, and adds a KL-projection step to restore constraint feasibility.
Key results
- On the gambler problem (threshold 2.5), RCPO satisfies the constraint throughout training while PCPO and RVI violate it, with reward close to RVI.
- On Frozen-Lake (threshold 0.7), only RCPO satisfies the constraint; PCPO and RVI violate it during training, and RCPO obtains more reward than PCPO.
- On Point Gather with Gaussian-noise perturbations, RCPO earns much higher reward than the non-robust CPO and PCPO while satisfying the utility constraint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RCPO instead of PCPO or RVI | Constrained RL · Gambler problem | constraint satisfaction during training | Helped |
| Use RCPO instead of RVI | Constrained RL · Gambler problem | reward | No clear effect |
| Use RCPO instead of PCPO | Constrained RL · N-chain | reward | Helped |
| Use RCPO instead of PCPO or RVI | Constrained RL · N-chain | constraint satisfaction | No clear effect |
| Use RCPO instead of PCPO or RVI | Constrained RL · Frozen-Lake | constraint satisfaction during training | Helped |
| Use RCPO instead of PCPO | Constrained RL · Frozen-Lake | reward | Helped |
| Use RCPO instead of CPO or PCPO under model uncertainty | Deep constrained RL · Point Gather | reward | Helped |
| Use RCPO under model uncertainty | Deep constrained RL · Point Gather | utility constraint satisfaction | Helped |
Limitations
- Tabular experiments use only 5 runs on small toy environments.
- Theoretical guarantees for continuous spaces incur an extra constant epsilon error from worst-case kernel approximation.
- Some approximations (e.g., Taylor expansions, Assumption on worst-case kernel accuracy) hold only for small step sizes.
Experimental setup
Datasets: gambler problem N-chain problem Frozen-Lake problem Point Gather (MuJoCo)
Benchmarks: gambler problem N-chain problem Frozen-Lake problem Point Gather
Metrics: robust reward robust utility (constraint satisfaction)
Baselines: PCPO RVI CPO
Builds on: CPO TRPO Robust value iteration PCPO
Closest papers by meaning
- SafeMPO: Constrained Reinforcement Learning with Probabilistic Incremental Improvement (ICLR 2026 · similarity 0.91)
- Reinforcement Learning under Model Mismatch (NeurIPS 2017 · similarity 0.91)
- Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning (AAAI 2022 · similarity 0.91)
- Robust Inverse Constrained Reinforcement Learning under Model Misspecification (ICML 2024 · similarity 0.91)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.90)
- Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.