Research map / Multi-agent systems and games / Games and equilibria
Can an AI Agent Safely Run a Government? Existence of Probably Approximately Aligned Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines a quantitative alignment metric for autonomous agents governing societies via social welfare theory, proves existence of computable probably approximately aligned (PAA) policies given an approximate world model with bounded KL error, and gives a method to make any black-box policy safe by action masking.
Problem
Autonomous agents making social decisions lack formal, a priori guarantees that their actions maximize societal welfare; existing alignment metrics are a posteriori and provide no safety guarantees.
Approach
Models social decision-making as a Social MDP whose reward is the expected power-mean social welfare of utilities; defines alignment as maximizing expected future discounted social welfare, and shows a modified sparse sampling planner using an approximate world model and sampled assessors yields PAA policies. Any black-box policy can be restricted to actions whose estimated value clears a safety t
Key results
- Proves existence of computable delta-epsilon-PAA policies when the world model KL error is below epsilon^2(1-gamma)^4/(8 Delta W^2).
- Shows any black-box policy can be made delta-omega-safe by restricting its action set using the planner's Q estimates, with no performance-optimality guarantee.
- Derives concentration bound for power-mean welfare under sampling without replacement, noting egalitarian and max rules require sample size linear in N.
Surprising findings
- Egalitarian (q=-infinity) and maximax (q=+infinity) welfare rules admit no sample-size-independent concentration guarantee, so their sample complexity degrades linearly with the society size.
Limitations
- Practical accuracy of the world model is hard to measure and sufficient KL accuracy may be unattainable, making PAA policies impractical for small epsilon.
- Results depend on the informational basis of utilities, a philosophical question outside the paper's scope.
- Assumes assessors can observe and evaluate entire social states and that preferences are static.
Experimental setup
Builds on: sparse sampling PAC learning Hoeffding-Serfling inequality action masking
Closest papers by meaning
- Policy Aggregation (NeurIPS 2024 · similarity 0.87)
- Position: Intent-aligned AI Systems Must Optimize for Agency Preservation (ICML 2024 · similarity 0.86)
- Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment (AAAI 2025 · similarity 0.86)
- AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.85)
- Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback (ICML 2024 · similarity 0.84)
- Calibrating Conservatism for Scalable Oversight (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.