Research map / Multi-agent systems and games / Human-AI and goal-conditioned
The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy
William Overman, Mohsen Bayati
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train AI and human by independent policy gradient on shared safety reward | wrapped unsafe base policy · Lavaland | violation rate: ~15% -> zero | Improved |
| train both players over time on shared reward | Oversight Game players · Lavaland | intervention rate: 50% -> 28.2% | Improved |
| relax team-game to unshared asymmetric costs | Oversight Game · Lavaland | violation rate | No clear effect |
| fine-tune LLM policies with independent policy gradient | Qwen3-30B-A3B LoRA agents · ToolEmu Oversight MDPs | test reward: -20 -> +8.4 | Improved |
| learned joint policy at risky vs safe states | LLM oversight policies · ToolEmu Oversight MDPs | AI ask rate: 99.4% at risky, 3.9% at safe | Improved |
Closest papers by meaning
- Calibrating Conservatism for Scalable Oversight (ICML 2026 · similarity 0.89)
- Optimizing Agent Planning for Security and Autonomy (ICLR 2026 · similarity 0.88)
- Position: Intent-aligned AI Systems Must Optimize for Agency Preservation (ICML 2024 · similarity 0.87)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.87)
- Can an AI Agent Safely Run a Government? Existence of Probably Approximately Aligned Policies (NeurIPS 2024 · similarity 0.87)
- The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
