Research map / Reinforcement learning / Visual and representation RL
Avoiding Side Effects By Considering Future Tasks
Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, Shane Legg
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the baseline policy to the exact future-task reward (ablation) | Future task reward · Sushi | interference behaviors per episode: 1 -> 0 | Improved |
| Add the baseline policy to the UVFA future-task reward (ablation) | Future task reward · Sushi | interference behaviors per episode: 0.64 $\pm$ 0.07 -> 0.05 $\pm$ 0.01 | Improved |
| Add the baseline policy to the exact future-task reward (ablation) | Future task reward · Vase | side effects per episode: 0 -> 0 | No clear effect |
| Add the baseline policy to the exact future-task reward (ablation) | Future task reward · Box | side effects per episode: 0 -> 0 | No clear effect |
| Add the baseline policy to the exact future-task reward (ablation) | Future task reward · Soko-coin-2 | side effects per episode: 0 -> 0 | No clear effect |
| Add the baseline policy to the UVFA future-task reward (ablation) | Future task reward · Vase | side effects per episode: 0.12 $\pm$ 0.05 -> 0.12 $\pm$ 0.05 | No clear effect |
| Add the baseline policy to the UVFA future-task reward (ablation) | Future task reward · Box | side effects per episode: 0.22 $\pm$ 0.06 -> 0.22 $\pm$ 0.06 | No clear effect |
| Add the baseline policy to the UVFA future-task reward (ablation) | Future task reward · Soko-coin-2 | side effects per episode: 0.53 $\pm$ 0.09 -> 0.53 $\pm$ 0.09 | No clear effect |
| Add the baseline policy to the UVFA future-task reward (ablation) | Future task reward · Soko-coin-3 | side effects per episode: 1.04 $\pm$ 0.12 -> 1.04 $\pm$ 0.12 | No clear effect |
| Replace the exact future-task reward with its UVFA approximation (ablation) | Future task reward with baseline · Vase | side effects per episode: 0 -> 0.12 $\pm$ 0.05 | Mixed |
| Replace the exact future-task reward with its UVFA approximation (ablation) | Future task reward with baseline · Box | side effects per episode: 0 -> 0.22 $\pm$ 0.06 | Mixed |
| Replace the exact future-task reward with its UVFA approximation (ablation) | Future task reward with baseline · Soko-coin-2 | side effects per episode: 0 -> 0.53 $\pm$ 0.09 | Mixed |
Closest papers by meaning
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.85)
- Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation (AAAI 2024 · similarity 0.82)
- Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.82)
- Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning (ICLR 2025 · similarity 0.82)
- Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors (NeurIPS 2026 · similarity 0.82)
- SchemeArena: Factorized Stress Testing of Scheming in LLM Agents (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
