Research map / Reinforcement learning / Policy gradient and actor-critic
Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Imitation Safe RL (SIM), a non-adversarial imitation method that imitates good trajectories and avoids bad ones (by reward threshold and cost constraint or oracle) to solve constrained RL without local cost constraints.
Problem
Constrained RL typically converts trajectory-level cost constraints into per-state surrogate cost constraints, which can cause over- or under-estimation of cost and thus conservative or aggressive policies; the paper targets safe RL that enforces trajectory-level constraints directly.
Approach
Label trajectories generated by the current policy as good (high reward, satisfying the cost limit) or bad (low reward or violating the cost limit), then train the policy via a non-adversarial distribution-matching objective that maximizes KL divergence between the mixed good/policy occupancy and the bad occupancy, with a discriminator-like ratio estimator K updated cooperatively; the good and bad
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Imitate good trajectories and avoid bad trajectories instead of using the initial policy alone | Initial policy π⁰ · SafetyGym (six environments) | expected reward while safe | Helped |
| Use only good demonstrations, removing bad demonstrations (ablation) | SIM · SafetyGym (two environments) | expected reward and safety | Hurt |
| Use only bad demonstrations, removing good demonstrations (ablation) | SIM · SafetyGym (two environments) | expected reward and safety | Hurt |
| Replace the non-adversarial update with two adversarial discriminators (ablation) | SIM · SafetyGym (two environments) | policy performance and stability | Hurt |
| Use GAIL with only good demonstrations (ablation) | GAIL · SafetyGym (two environments) | expected reward and safety | Hurt |
| Use GAIL with only bad demonstrations (ablation) | GAIL · SafetyGym (two environments) | expected reward and safety | Hurt |
| Use GAIL with both good and bad demonstrations and two discriminators (ablation) | GAIL · SafetyGym (two environments) | expected reward and safety | Hurt |
| Use BC with one good demonstration instead of SIM | SIM · Two SafetyGym environments | expected reward while satisfying the constraint | Hurt |
| Use BC with both good and bad demonstrations instead of SIM | SIM · Two SafetyGym environments | expected reward while satisfying the constraint | Hurt |
| Start with an entry-level or medium-level pre-trained policy rather than an expert-level one | SIM · SafetyCarButton and SafetyCarPush | expected reward | Hurt |
| Improve an entry-level or medium-level initial policy with SIM rather than leaving it as PPO-Lagrangian | PPO-Lagrangian · SafetyCarButton and SafetyCarPush | expected reward | Helped |
| Apply SIM using an oracle to identify constraint-violating trajectories when the cost function is unknown | Unconstrained initial policy · SafetyPointGoal and SafetyCarPush | safety with competitive expected reward | Helped |
Surprising findings
- Learning from bad trajectories alone does not yield a good policy, per the authors' experiments.
- Initial policy trained with a relaxed cost limit (28) yields more useful good trajectories for SIM.
Limitations
- Fails when feasible trajectories are hard to generate under strict constraints, since good demonstrations are needed.
- Performance depends on the expertise of the initial policy; weaker initial policies give lower returns.
Experimental setup
Datasets: Safety-Gym (SafetyPointGoal, SafetyCarGoal, SafetyPointButton, SafetyCarButton, SafetyPointPush, SafetyCarPush) Mujoco Circle Mujoco Velocity
Benchmarks: Safety-Gym Mujoco Circle Mujoco Velocity
Metrics: expected return expected cost CVaR cost
Baselines: PPO-Lagrangian FOCOPS CUP CPO PPO-unconstrained BC BC-GB GAIL variants
Builds on: PPO PPO-Lagrangian GAIL IQ-learn
Closest papers by meaning
- Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.94)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.92)
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.92)
- Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · similarity 0.91)
- From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.