Research map / Reinforcement learning / RL from feedback
e-COP : Episodic Constrained Optimization of Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes e-COP, the first policy optimization algorithm for constrained RL in episodic (finite-horizon) MDPs, built on a new episodic policy difference lemma and a quadratic-damped Lagrangian.
Problem
Constrained reinforcement learning in episodic (finite-horizon) settings, where existing policy optimization methods are designed for infinite-horizon discounted tasks and struggle with constraint satisfaction and optimality.
Approach
Derives a policy difference lemma for the episodic setting and solves a per-timestep constrained policy update via a quadratic-damped Lagrangian (augmented-Lagrangian-style penalty with lagged multipliers), avoiding matrix inversions and giving stable constraint control.
Key results
- Reports superior empirical performance over adapted infinite-horizon methods PPO, CPO, TRPO and IPO on MuJoCo episodic constrained tasks (Ant Gather, Bottleneck, Grid).
- Demonstrates multiple-constraint satisfaction in a Navigation environment with hazard and pillar costs.
- Ablation shows a pre-trained e-COP agent evaluated under different cost thresholds on Humanoid and Point Circle.
Limitations
- Only the poster body is available; numerical results are shown only as figures without reported values.
- Experiments are limited to MuJoCo-based navigation and locomotion environments.
Experimental setup
Benchmarks: Ant Gather Bottleneck Grid Humanoid Point Circle Navigation (hazards and pillars)
Metrics: episodic reward episodic constraint value
Baselines: PPO CPO TRPO IPO
Builds on: PPO CPO TRPO IPO
Closest papers by meaning
- Episodic Policy Gradient Training (AAAI 2022 · similarity 0.89)
- Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.86)
- Online Reinforcement Learning in Non-Stationary Context-Driven Environments (ICLR 2025 · similarity 0.86)
- Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.