Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Sim-to-Real Causal Transfer: A Metric Learning Approach to Causally-Aware Interaction Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits the CausalAgents benchmark for causal awareness in multi-agent trajectory forecasting, builds an ORCA counterfactual diagnostic dataset, and proposes contrastive and ranking causal regularizers plus sim-to-real cross-domain multi-task transfer.
Problem
Trajectory forecasting and crowd models often lack causal awareness of neighboring agents' influence; the existing CausalAgents benchmark has annotation and evaluation caveats (missed indirect effects, group removal), and real-world causal annotations are scarce.
Approach
Regularize the latent representation so that counterfactual embeddings (with an agent removed) are separated by a distance proportional to that agent's causal effect, using either a contrastive loss on binary causal labels or a margin ranking loss on continuous causal effects; transfer causal knowledge from ORCA simulation to real data via joint multi-task training.
Key results
- Ranking-based causal regularization lowers ADE/FDE on NBA rebound from 0.562/1.271 (AutoBots baseline) to 0.544/1.235 m.
- Recent models show ACE-NC errors roughly 10x smaller than ACE-DC/IC, indicating partial robustness to non-causal removal but underestimation of indirect causal effects.
- On ETH-UCY, the causal transfer method with only 25% real data outperforms AutoBots baseline; with 50% real data it surpasses all compared models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add causal contrastive regularization | AutoBots · NBA rebound | ADE: 0.562 -> 0.550 | Helped |
| Add causal contrastive regularization | AutoBots · NBA rebound | FDE: 1.271 -> 1.248 | Helped |
| Add causal ranking regularization | AutoBots · NBA rebound | ADE: 0.562 -> 0.544 | Helped |
| Add causal ranking regularization | AutoBots · NBA rebound | FDE: 1.271 -> 1.235 | Helped |
| Use vanilla sim-to-real transfer without causal regularization | AutoBots · NBA rebound | ADE: 0.562 -> 0.567 | Hurt |
| Use non-causal data augmentation | AutoBots · NBA rebound | ADE: 0.562 -> 0.569 | Hurt |
| Add causal regularization (contrastive or ranking) | AutoBots · Diagnostic dataset | ACE | Helped |
| Use fine-grained causal-effect annotations with ranking regularization | AutoBots · Diagnostic dataset | ACE | Helped |
| Add causal ranking regularization | AutoBots · OOD-Context and OOD-Density | prediction error | Helped |
| Transfer simulation causal supervision to real-world training | AutoBots · ETH-UCY | prediction error | Helped |
| Add causal regularizers to sim-to-real transfer | AutoBots · ETH-UCY | prediction accuracy | Helped |
| Use vanilla sim-to-real transfer rather than causal transfer | AutoBots · ETH-UCY low-data regimes | prediction error | Hurt |
Surprising findings
- Recent representations underestimate causal effects, especially indirect ones, rather than overestimating non-causal agent influence as the CausalAgents benchmark suggested.
- Vanilla sim-to-real transfer (combining synthetic and real data) struggles in low-data regimes and can hurt performance.
Limitations
- Supervision alone is likely insufficient for complex causal reasoning; indirect effects remain underestimated.
- Causal annotations are only available in simulation or for limited benchmarks; real-world causal labels are absent.
- Causal evaluation (ACE) only possible where ground-truth causal effects are available from controlled simulation.
Experimental setup
Datasets: ETH-UCY NBA rebound ORCA simulated diagnostic dataset CausalAgents (Waymo annotations)
Benchmarks: ORCA diagnostic dataset ETH-UCY NBA rebound
Metrics: ADE FDE ACE ACE-NC ACE-DC ACE-IC
Baselines: AutoBots D-LSTM S-LSTM Trajectron++ STGCNN Multi-Transmotion Non-causal data augmentation SimAug vanilla sim-to-real
Builds on: CausalAgents AutoBots SimAug Social NCE ORCA
Closest papers by meaning
- CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations (NeurIPS 2025 · similarity 0.88)
- Learning Linear Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity (NeurIPS 2024 · similarity 0.87)
- Curious Causality-Seeking Agents Learn Meta Causal World (NeurIPS 2025 · similarity 0.87)
- Tackling Non-Stationarity in Reinforcement Learning via Causal-Origin Representation (ICML 2024 · similarity 0.87)
- Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains (ICML 2025 · similarity 0.87)
- On the Identification of Temporal Causal Representation with Instantaneous Dependence (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.