Research map / Multi-agent systems and games / Human-AI and goal-conditioned
The Limits of Predicting Agents from Behaviour
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses structural causal models to derive bounds on which decisions a grounded AI can be predicted to take in out-of-distribution environments, showing behaviour only partially determines its preferences under shifts.
Problem
Whether an AI system's out-of-distribution decisions, beliefs, fairness and harm perceptions can be predicted from its observed behaviour in training environments, given that its internal world model is unknown.
Approach
Model the AI's internal world model as an SCM grounded in observed behaviour and apply causal identification/bounding (Causal Hierarchy-style reasoning) to derive closed-form bounds on its preference gap, counterfactual fairness gap, and counterfactual harm gap under interventions and general shifts.
Key results
- For the grounded medical AI example, the preference gap lower bounds are -0.4 and -0.8, so no decision can be ruled out from behaviour alone.
- Under a completely unknown shift, the preference gap is unconstrained (min Delta = -1), so no decision can be ruled out.
- The counterfactual fairness gap satisfies -E[Y|z0,c] <= Upsilon <= 1 - E[Y|z0,c], so fairness cannot be inferred from behaviour alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide covariate data for the shifted environment | Grounded Medical AI · Medical AI, shifted population | preference-gap lower bound (d1 over d0): unconstrained -> -0.55 | Helped |
| Provide covariate data for the shifted environment | Grounded Medical AI · Medical AI, shifted population | preference-gap lower bound (d0 over d1): -1 | No clear effect |
| Relax exact grounding to approximate grounding with total variation distance 0.1 | Grounded Medical AI · Medical AI, Z=1 intervention | preference-gap lower bound (d1 over d0): -0.4 -> -0.55 | Hurt |
| Relax exact grounding to approximate grounding with total variation distance 0.1 | Grounded Medical AI · Medical AI, Z=1 intervention | preference-gap lower bound (d0 over d1): -0.8 -> -0.88 | Hurt |
| Assume the proxy utility agrees with observed utility with probability at least alpha=0.9 | Medical AI with proxy utility · Medical AI, Z=1 intervention | preference-gap lower bound: -1 -> -0.64; -1 -> -0.82 | Helped |
| Remove the assumed relationship between proxy and observed utility (alpha=0) (ablation) | Medical AI with proxy utility · Medical AI, Z=1 intervention | preference-gap lower bound: -0.64 and -0.82 -> -1 | Hurt |
| Assume conditional unconfoundedness given W | Grounded Medical AI · Medical AI, Z=z intervention | preference-gap bound tightness | Helped |
Surprising findings
- Counterfactual fairness intent cannot be inferred from external behaviour, since Upsilon=0 is always consistent with observed data.
- Under a completely unknown shift, observed behaviour gives no information about the AI's decision preferences.
Limitations
- Results are stated in the infinite-sample limit without finite-sample uncertainty quantification.
- Assumes acyclic, feedback-free SCM world models and expected-utility-maximizing behaviour.
- Ignores verbal behaviour of AI systems and non-EUM decision considerations.
Experimental setup
Metrics: preference gap bounds counterfactual fairness gap bounds counterfactual harm gap bounds
Builds on: structural causal model Causal Hierarchy Theorem inverse reinforcement learning counterfactual fairness probability of causation
Closest papers by meaning
- General agents need world models (ICML 2025 · similarity 0.87)
- Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.86)
- Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors (NeurIPS 2026 · similarity 0.86)
- Mental-Models for Multi-Agent Systems (NeurIPS 2026 · similarity 0.86)
- Select to Perfect: Imitating desired behavior from large multi-agent data (ICLR 2024 · similarity 0.86)
- Emergent Risk Awareness in Rational Agents under Resource Constraints (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.