Research map / Large language models / AI and society

A Black Swan Hypothesis: The Role of Human Irrationality in AI Safety

ICLR 2025 · Large language models › AI and society · approach: Human-centered trust and agency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the "s-black swan" concept: high-risk, rare state-action events that arise from human misperception of reward and likelihood under Cumulative Prospect Theory even in stationary MDPs, with theoretical formalization and bounds.

Problem

Black swan events are typically attributed to non-stationary environments, but the community lacks a comprehensive definition; the paper asks whether high-risk, rare events can arise in static environments due to human misperception of rewards and probabilities.

Approach

Models human misperception using Cumulative Prospect Theory's value and probability distortion functions applied to MDP rewards and visitation probabilities, introduces Human MDP and Human-Estimation MDP, and defines s-black swans as state-action pairs perceived as impossible yet rare and high-risk in the ground-truth stationary MDP.

Key results

• Theorem: with |S|=2 or T=1 the optimal policy under the human-distorted MDP coincides with the ground-truth optimal policy, while a three-state two-step counterexample exists.
• Theorem: the value perception gap between the Human MDP and ground-truth MDP has a non-zero lower bound even with zero estimation error, growing with reward distortion C_bs and minimum s-black swan probability.
• Hitting-time theorem gives the number of steps t needed to encounter an s-black swan with probability at least delta.

Surprising findings

• With bandit (T=1) or two-state MDPs, human reward and probability distortion does not change the optimal policy.
• Value perception gap has a non-zero lower bound regardless of horizon length, even though value estimation error converges to zero.

Limitations

• Purely theoretical; no empirical experiments validating the s-black swan framework on real or simulated decision-making.
• Lehman Brothers and other case examples are argued heuristically rather than established as s-black swans.
• Relies on assumed CPT distortion functions and a relative convexity assumption.

Experimental setup

Builds on: Cumulative Prospect Theory Expected Utility Theory safe reinforcement learning criteria

Closest papers by meaning

• Position: AI Safety Must Embrace an Antifragile Perspective (ICML 2025 · similarity 0.85)
• Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor (NeurIPS 2025 · similarity 0.81)
• The Limits of Predicting Agents from Behaviour (ICML 2025 · similarity 0.81)
• Reasoning About Actual Causes in Nondeterministic Domains (AAAI 2025 · similarity 0.80)
• Regulating AI: Applying Insights from Behavioural Economics and Psychology to the Application of Article 5 of the EU AI Act (AAAI 2024 · similarity 0.80)
• Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.80)