Research map / Multi-agent systems and games / Games and equilibria
The Partially Observable Off-Switch Game
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Partially Observable Off-Switch Game (PO-OSG), a game-theoretic model of AI shutdown under asymmetric information, and proves that partial observability and bounded communication can make AIs avoid shutdown or defer less, even when humans are perfectly rational.
Problem
The off-switch game assumes the human fully observes the environment, but in realistic settings humans have limited information and AIs may hold private information, so it is unclear when AIs defer to humans on shutdown.
Approach
Extends the off-switch game to partial observability and cheap-talk communication, using common-payoff dynamic Bayesian games, and shows that giving the human more information or the AI less (or adding communication) can make the AI wait less often in optimal play.
Key results
- In the File Deletion Game, the optimal policy pair has expected payoff 7/4, exceeding the best always-wait policy pair's payoff of 1, so the AI can avoid deferring in optimal play.
- Making the human strictly more informed (full version number vs first digit) in a three-state variant causes the AI to wait strictly less often in optimal policy pairs.
- Communication with a larger message set for the AI can make the AI wait strictly less often, and a single extra message can shift waiting probability to arbitrarily low levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Have Rob act on legacy code rather than always wait | PO-OSG in the File Deletion Game · File Deletion Game | expected payoff: +1 -> +7/4 | Helped |
| Give the human the full version number instead of only its first digit | PO-OSG in the File Deletion Game · File Deletion Game | expected payoff: 2/3 -> 4/3 | Helped |
| Give the human the full version number instead of only its first digit | PO-OSG in the File Deletion Game · File Deletion Game | Rob deferral | Hurt |
| Make Rob's observations strictly less informative | PO-OSG · PO-OSG optimal policy pairs | Rob deferral | Hurt |
| Allow Rob a more expressive message system | PO-OSG-C · PO-OSG-C optimal policy pairs | Rob deferral | Hurt |
| Reduce the human's message-system expressiveness | PO-OSG-C · PO-OSG-C optimal policy pairs | Rob deferral | Hurt |
| Increase the informativeness of either agent's observations using coordinated garbling comparisons | PO-OSG · PO-OSG optimal policy pairs | expected payoff | Helped |
| Increase communication bandwidth | PO-OSG-C · PO-OSG-C optimal policy pairs | expected payoff | Helped |
| Use Structure 2 rather than Structure 1, with an uncoordinated garbling between structures | PO-OSG garbling example · Garbling example | expected payoff: \frac{3}{4} -> 1 | Helped |
| Give a Rob-unaware human more informative observations | PO-OSG with a Rob-unaware human · Rob-unaware optimal policy pairs | expected payoff | Hurt |
Surprising findings
- Making the human more informed or the AI less informed can cause the AI to defer less in optimal play.
- Bounded communication expansion can decrease AI deference, and making the human's observations less informative can increase payoffs for the pair in some cases.
- Value of information is not necessarily positive when the human is unaware of the AI (rob-unaware human).
Limitations
- Assumes costless human feedback, single-round interaction, common payoffs, and a rational human.
- Analysis covers optimal policy pairs; algorithms that fail to find optimal solutions may behave differently.
- Unclear whether the counterintuitive examples arise frequently in real-world settings.
Experimental setup
Metrics: expected common payoff under optimal policy pairs probability of waiting (deferral)
Baselines: Off-Switch Game
Builds on: Off-Switch Game Cooperative Inverse Reinforcement Learning assistance games Blackwell informativeness
Closest papers by meaning
- Observation Interference in Partially Observable Assistance Games (ICML 2025 · similarity 0.88)
- State-Constrained Zero-Sum Differential Games with One-Sided Information (ICML 2024 · similarity 0.84)
- Is Knowledge Power? On the (Im)possibility of Learning from Strategic Interactions (NeurIPS 2024 · similarity 0.83)
- Playing Markov Games Without Observing Payoffs (NeurIPS 2026 · similarity 0.83)
- Explaining Decisions of Agents in Mixed-Motive Games (AAAI 2025 · similarity 0.83)
- Solving Hierarchical Information-Sharing Dec-POMDPs: An Extensive-Form Game Approach (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.