Research map / Multi-agent systems and games / Human-AI and goal-conditioned

Robust agents learn causal world models

ICLR 2024 · Multi-agent systems and games › Human-AI and goal-conditioned · approach: Intrinsically motivated exploration · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically that an agent whose policies satisfy a regret bound across a large set of local distributional shifts must have learned an approximate causal Bayesian network of the data generating process, with the approximation exact for optimal agents.

Problem

Whether agents must learn causal models of the data generating process in order to generalise (achieve low regret) under distributional shifts, or whether other inductive biases suffice.

Approach

Proves an equivalence between identifying the underlying causal Bayesian network and having regret-bounded optimal policies across mixtures of local interventions; the policy oracle over shifts reveals the causal model, so robust adaptation necessitates a causal model (a model-independent, causal good regulator-style result).

Key results

• Theorem: for almost all CIDs, optimal policies under mixtures of local interventions identify the underlying causal DAG and joint distribution over ancestors of utility.
• Approximate theorem: parameter error of recovered causal model bounded by a function gamma(delta) that is zero at zero regret and grows linearly for small regret.
• Experiments on randomly generated CIDs show learned DAG error and P(x,y) error increase with the regret bound.

Surprising findings

• Optimal policies under domain shifts for a single utility encode the full causal model (all L2 queries), so learning robust policies is equivalent to learning a causal model.
• Learning causal structure over latent variables is possible by observing only a single decision variable's response to interventions, without measuring many variables.

Limitations

• Results require robustness to a large set of local interventions over all environment variables; loosening regret bounds makes some causal relations unidentifiable.
• Only applies to unmediated decision tasks; mediated tasks such as MDPs not covered by theorems.
• Results hold for almost all CIDs (Lebesgue measure zero exceptions) and assume causal sufficiency and categorical variables.

Experimental setup

Datasets: randomly generated causal influence diagrams (synthetic)

Metrics: error rate of learned DAG vs regret bound mean error in P(x,y) vs regret bound

Baselines: causal transportability theory causal hierarchy theorem

Builds on: causal influence diagrams transportability theory causal hierarchy theorem causal representation learning

Closest papers by meaning

• Agents Robust to Distribution Shifts Learn Causal World Models Even Under Mediation (NeurIPS 2025 · similarity 0.91)
• Teaching Transformers Causal Reasoning through Axiomatic Training (ICML 2025 · similarity 0.86)
• Causal Strategic Learning with Competitive Selection (AAAI 2024 · similarity 0.85)
• General agents need world models (ICML 2025 · similarity 0.85)
• Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? (NeurIPS 2024 · similarity 0.85)
• Causal Bandits: The Pareto Optimal Frontier of Adaptivity, a Reduction to Linear Bandits, and Limitations around Unknown Marginals (ICML 2024 · similarity 0.85)