Research map / Large language models / AI and society

Towards a Theory of AI Personhood

AAAI 2025 · Large language models › AI and society · approach: Human-centered trust and agency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A position/theory paper that proposes three necessary conditions for AI personhood (agency, theory-of-mind, self-awareness), reviews mixed ML evidence on contemporary LMs, and discusses implications for alignment and ethical treatment.

Problem

When, if ever, should AI systems be ascribed personhood, and how does this bear on alignment and the ethical treatment of AI?

Approach

Decomposes AI personhood into agency (intentional-stance-based mental states and robust adaptation), theory-of-mind (higher-order intentional states and communicative language use), and self-awareness (self-knowledge, self-location, introspection, self-reflection), then argues that self-reflection on goals may induce goal change and undermine control-based alignment framings.

Key results

• Reviews evidence on contemporary LMs for agency, theory-of-mind, and self-awareness and finds it surprisingly inconclusive overall.
• Notes SOTA LMs perform significantly worse than human baselines but better than chance on self-knowledge benchmarks, and that fine-tuning for interaction improves performance.
• Argues no existing work evaluates self-reflection on goals in AI systems.

Surprising findings

• Self-reflection could cause AI goals to change, a neglected point in AI risk considerations, which may make control-based alignment framings untenable if AI systems are persons.

Limitations

• Explicitly takes no confident philosophical stance; conditions are contested.
• Evidence review is drawn from literature rather than new experiments.
• Identity and embodiment are acknowledged as unresolved components of personhood.

Experimental setup

Builds on: intentional stance Grice's theory of communicative meaning legal personhood

Closest papers by meaning

• Position: Intent-aligned AI Systems Must Optimize for Agency Preservation (ICML 2024 · similarity 0.88)
• Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.87)
• AI Agents Push Humans Out of the Loop (NeurIPS 2026 · similarity 0.86)
• Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment (AAAI 2025 · similarity 0.86)
• Position: A Roadmap to Pluralistic Alignment (ICML 2024 · similarity 0.86)
• Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties (AAAI 2024 · similarity 0.86)