Research map / Large language models / AI and society

Position: Humanity Faces Existential Risk from Gradual Disempowerment

ICML 2025 · Large language models › AI and society · approach: Human-centered trust and agency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing that incremental AI advances could gradually erode human influence over the economy, culture, and states, potentially causing an irreversible existential catastrophe without any single abrupt takeover.

Problem

Existing AI existential risk discussion centers on abrupt misalignment or takeover and misuse; the paper examines whether gradual, incremental AI progress could cause permanent disempowerment of humanity through erosion of human influence over large-scale societal systems.

Approach

Introduces "gradual disempowerment": societal systems are aligned with human interests only because they rely on human labor and cognition and on explicit human control; AI replacing that reliance removes implicit alignment, and interlinked economic, cultural and state systems can reinforce misalignment, without any acute capability jump.

Key results

• Argues that six linked claims (implicit alignment via human labor/cognition, interdependence of societal systems, correlated misalignment) can jointly lead to permanent human disempowerment without any single abrupt AI takeover.
• Analyzes incentives for AI adoption in economy, culture and states, distinguishing relative disempowerment from absolute disempowerment in each domain.
• Proposes intervention categories: measuring disempowerment, preventing excessive AI influence, strengthening human influence, and system-wide alignment.

Surprising findings

• Argues that even a human-aligned AGI may itself be marginalized by competitive dynamics, so aligning individual AI systems may be insufficient.

Limitations

• No empirical evaluation; arguments are conceptual and qualitative.
• Mitigation proposals are acknowledged as likely to serve mostly as stopgaps.
• Proposed research agenda (e.g., simulating civilizations with LLMs) is not carried out.

Experimental setup

Builds on: Christiano 2019 'get what we can measure' slow catastrophe Critch & Russell 2023 societal-scale risk taxonomy Kasirzadeh accumulative AI x-risk hypothesis Korinek & Stiglitz 2018 Malthusian dynamics

Closest papers by meaning

• Position: AI Safety should prioritize the Future of Work (ICML 2025 · similarity 0.86)
• Position: Evolving AI Collectives Enhance Human Diversity and Enable Self-Regulation (ICML 2024 · similarity 0.84)
• Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs (NeurIPS 2025 · similarity 0.84)
• Regulating AI: Applying Insights from Behavioural Economics and Psychology to the Application of Article 5 of the EU AI Act (AAAI 2024 · similarity 0.84)
• Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback (ICML 2024 · similarity 0.83)
• Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment (AAAI 2025 · similarity 0.83)