Research map / Reinforcement learning / Policy gradient and actor-critic
Constrained policy optimization
Solve constrained MDPs with Lagrangian-style methods, handling bias, exploration, and model mismatch.
48 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Policy gradient and actor-critic, keyword cluster “safe · constrained · constraints”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
13 papers in 2023–24 and 20 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.8% to 4.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Constrained Policy Optimization (ICML 2017 · 109 citations)
- 2019 · Reward Constrained Policy Optimization (ICLR 2019 · 254 citations)
- 2020 · IPO: Interior-Point Policy Optimization under Constraints (AAAI 2020 · 134 citations)
- 2021 · Balancing Constraints and Rewards with Meta-Gradient D4PG (ICLR 2021 · 6 citations)
- 2022 · Reachability Constrained Reinforcement Learning (ICML 2022 · 6 citations)
- 2023 · Gradient-Adaptive Pareto Optimization for Constrained Reinforcement Learning (AAAI 2023 · 7 citations)
- 2024 · Enhancing Off-Policy Constrained Reinforcement Learning through Adaptive Ensemble C Estimation (AAAI 2024 · 5 citations)
- 2025 · CHPO: Constrained Hybrid-action Policy Optimization for Reinforcement Learning (NeurIPS 2025 · 1 citation)
- 2026 · Proactive Constrained Policy Optimization with Preemptive Penalty (AAAI 2026 · 1 citation)
Earliest papers here
- Constrained Policy Optimization (ICML 2017 · 109 citations)
- Reward Constrained Policy Optimization (ICLR 2019 · 254 citations)
- Reinforcement Learning with Convex Constraints (NeurIPS 2019 · 48 citations)
What has been tried
35 papers here report 305 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 8%, hurt 19%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Remove adaptive ensemble C-learning | Walker2d-v3 and Swimmer-v3 · Reward | Hurt | Enhancing Off-Policy Constrained Reinforcement Learning through Adaptive Ensemble C Estimation (AAAI 2024 · 5 citations) | |
| combine CAGrad with the Lagrangian approach instead of using CoMOGA | Quadrupedal task · constraint satisfaction | Hurt | Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning (ICLR 2025 · 1 citation) | |
| Use an appropriate reward penalty λ rather than a small λ or λ = 0 | GridWorld · expected reward | Helped | Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · 1 citation) | |
| Add de-biased initialization | Synthetic benchmark · average episode reward | Helped | Autoregressive Policy Optimization for Constrained Allocation Tasks (NeurIPS 2024 · 1 citation) | |
| Remove the Lagrangian multiplier | Portfolio · performance | 100% -> 96% | Hurt | Augmented Lagrangian Risk-constrained Reinforcement Learning for Portfolio Optimization (Student Abstract) (AAAI 2025 · 0 citations) |
| Add the ||a-hat||_2^2 entropy term | Action-constrained environments · constraint violations (%) | Helped | Leveraging Constraint Violation Signals for Action Constrained Reinforcement Learning (AAAI 2025 · 0 citations) | |
| add the EVT-based constraint objective instead of a constant quantile constraint objective | Safety Gymnasium and Safety MuJoCo · policy performance | Helped | Extreme Value Policy Optimization for Safe Reinforcement Learning (ICML 2025 · 0 citations) | |
| Replace memory-driven intrinsic cost with a fixed constant | Safety Gym and Safety MuJoCo · policy performance | Hurt | Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration (ICML 2025 · 0 citations) | |
| Set recovery parameter t to 0.75 instead of using the cost-only recovery scheme (t=1) | OpenAI Gym tasks · reward and constraint performance | Helped | ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints (ICML 2024 · 0 citations) | |
| add the constraint | FetchPush · Success Rate (%) | 78.3 (5.7) -> 92.7 (3.1) | Helped | QPRL : Learning Optimal Policies with Quasi-Potential Functions for Asymmetric Traversal (ICML 2025) |
| Use multi-objective SAC with preferences learned simultaneously instead of single-objective SAC with a fixed preference | HalfCheetah and Ant · forward reward and valid action rate | Helped | Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs (ICLR 2025) |
What reviewers will expect
Datasets and baselines named in the 15 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- PPO · 2 of 15 papers
- DPre+ · 2 of 15 papers
- SPre+ · 2 of 15 papers
- FlowPG · 2 of 15 papers
- CPO · 2 of 15 papers
- Saute · 2 of 15 papers
- Simmer · 2 of 15 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 33 papers; from OpenAlex and the papers' affiliations).
- Singapore Management University · 4 papers
- Shanghai Jiao Tong University · 2 papers
- Politecnico di Milano · 2 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Safety layers and shielding (85 papers)
- Safe policy improvement (27 papers)
Elsewhere in the topic
- Model-based RL with guarantees (245 papers)
- Regularized scalable deep value learning (216 papers)
- Temporal-difference and discounting theory (143 papers)
- Tree search and policy optimization (109 papers)
- Variance-reduced policy gradients (79 papers)
- Actor-critic theory (76 papers)
- Off-policy actor-critic with expressive policies (71 papers)
- Trajectory reuse in policy gradients (68 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).