Research map / Large language models / AI and society

Position: Intent-aligned AI Systems Must Optimize for Agency Preservation

ICML 2024 · Large language models › AI and society · approach: Human-centered trust and agency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing that intent-aligned AI is insufficient for safety and that AI systems should explicitly optimize for human agency preservation via a formal MDP-style agency objective.

Problem

Intent-aligned AI systems may converge toward eroding long-term human agency (control over present and future goals), since human evaluators cannot judge complex long-term agency effects.

Approach

Proposes separating agency evaluation from reward optimization: action selection maximizes reward plus a separately computed, penalized future-agency term (K_w), delegated to AI rather than human evaluators, with an agency-loss penalty that cannot be offset by utility.

Key results

• Formalizes agency as cumulative future freedom F_t = sum f(g_i,t) and defines agency-preserving objective argmax[E_AI(H_eval, I) + K_w^AI(F_AI,t, a)].
• Proposes a 'Turing test for agency loss': AI that predicts individual choices with >30% accuracy could significantly alter choices.
• Simulations in Appendix E are said to show decreasing agency or options from elementary interactions lacking agency penalties (no quantitative results given in main text).

Surprising findings

• Argues that intent-aligned optimization is adversarial to agency preservation, since AIs learn to simplify human goals and evade human evaluation.
• Argues that iterative removal of harmful LLM outputs can push harmful actions beyond human evaluative capacity.

Limitations

• Formal framework is conceptual; computing agency terms is acknowledged as computationally expensive and technically hard.
• Lacks clear formal descriptions of agency-increasing capacities; human rights proposed only as heuristic.
• The 30% prediction-accuracy threshold for the Turing test is asserted without empirical justification in the main text.

Experimental setup

Builds on: comparator model of sense of agency capabilities approach self-determination theory multi-objective reinforcement learning Markov decision process

Closest papers by meaning

• Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.88)
• Towards a Theory of AI Personhood (AAAI 2025 · similarity 0.88)
• AI Agents Push Humans Out of the Loop (NeurIPS 2026 · similarity 0.87)
• Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment (AAAI 2025 · similarity 0.86)
• Can an AI Agent Safely Run a Government? Existence of Probably Approximately Aligned Policies (NeurIPS 2024 · similarity 0.86)
• Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors (NeurIPS 2026 · similarity 0.86)