Research map / Reinforcement learning / Visual and representation RL
Avoiding Side Effects in Complex Environments
Alex Turner, Neale Ratzlaff, Prasad Tadepalli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AUP's penalty based on a learned auxiliary reward | PPO on append-spawn · append-spawn | reward: 111% of the reward | Improved |
| Add AUP's penalty based on a learned auxiliary reward | PPO on append-spawn · append-spawn | side effects: 39% of the side effects | Improved |
| Replace the CB-VAE auxiliary reward with a random projection (ablation) | AUP on append-spawn · append-spawn | reward: matches AUP on reward | No clear effect |
| Replace the CB-VAE auxiliary reward with a random projection (ablation) | AUP on append-spawn · append-spawn | side effects: 48% of PPO's side effects | Improved |
| Add AUP's penalty based on a learned auxiliary reward | PPO on append-still-easy · append-still-easy | side effects: 27.8% the side effects of PPO | Improved |
| Use a one-dimensional CB-VAE latent space (ablation) | AUP on append-still-easy · append-still-easy | episodic return: 98% of PPO's episodic return | No clear effect |
| Increase the CB-VAE latent dimension (ablation) | AUP · append-still-easy | side effect score | Hurt |
| Increase the number of auxiliary reward functions beyond 2 (ablation) | AUP · append-still-easy | side effects | Mixed |
| Increase the final penalty coefficient lambda* to .5 or higher (ablation) | AUP · append-still-easy | movement: lambda*=.5 | Hurt |
| Penalize L1 distance from the initial state instead of using AUP (ablation) | PPO on append-still-easy · append-still-easy | reward | Hurt |
| Add AUP's penalty based on a learned auxiliary reward | PPO on append-still · append-still | reward | Hurt |
| Add AUP's penalty based on a learned auxiliary reward | PPO on prune-still-easy · prune-still-easy | reward | No clear effect |
Closest papers by meaning
- Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation (AAAI 2024 · similarity 0.81)
- Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning (ICLR 2025 · similarity 0.80)
- Improving Regret Approximation for Unsupervised Dynamic Environment Generation (NeurIPS 2025 · similarity 0.80)
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.80)
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation (NeurIPS 2024 · similarity 0.79)
- Marginal Benefit Driven RL Teacher for Unsupervised Environment Design (AAAI 2025 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
