Research map / Large language models / Agents, code and math
COLD: Causal reasOning in cLosed Daily activities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COLD, a closed-system causal commonsense reasoning benchmark built from DeScript daily-activity scripts that yields ~9.5M causal query instances, and evaluates open-weight LLMs via MCQA prompting and backdoor-adjusted average treatment effect estimates.
Problem
Existing causal commonsense benchmarks are grounded in real-world events but lack causal graphs and exhaustive query coverage, while symbolic causal benchmarks lack real-world grounding; LLM causal reasoning about daily activities is hard to evaluate rigorously.
Approach
Use crowd-sourced script knowledge (DeScript observational DAGs of events per activity) plus manually built causal graphs to generate causal query triplets via d-separation, scaled by text paraphrase instances, and estimate causal strength with backdoor-adjusted ATE computed from LM temporal/intervention logits.
Key results
- Phi-2 reaches 85.14% success on the cake causal triplets under prompt-based MCQA, while gpt-neo-125M is near chance (50.71%).
- Backdoor-adjusted ΔM with gemma-7b reaches 72.80% on cake and 77.80% on shopping, improving over temporal-scheme estimates.
- Five graduate students averaged 92.20% on 100 sampled triplets, versus lower LLM results on the bus and tree activities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use backdoor adjustments instead of temporal estimation with gemma-7b | Causal-strength estimation on COLD · Cake | accuracy: 46.8 -> 72.80 | Helped |
| Use backdoor adjustments instead of temporal estimation with gemma-7b | Causal-strength estimation on COLD · Shopping | accuracy: 54.00 -> 77.80 | Helped |
| Use backdoor adjustments instead of temporal estimation with Mistral-7B-v0.1 | Causal-strength estimation on COLD · Train | accuracy: 64.50 -> 73.70 | Helped |
| Use backdoor adjustments instead of temporal estimation with gemma-7b | Causal-strength estimation on COLD · Tree | accuracy: 52.70 -> 71.90 | Helped |
| Use backdoor adjustments instead of temporal estimation with Mistral-7B-v0.1 | Causal-strength estimation on COLD · Bus | accuracy: 62.30 -> 67.00 | Helped |
| Evaluate with temporally plausible but causally implausible incorrect choices | LLMs evaluated on COLD causal triplets · COLD | success rate | Hurt |
Surprising findings
- Temporal-ordering logit differences serve as a surprisingly good proxy for causal strength, and backdoor adjustment substantially improves over the temporal scheme for many models.
- Several small models (gpt-neo-125M, gpt-neo-2.7B, gpt-j-6B) perform at chance on prompt-based causal queries.
Limitations
- Only five daily activities are covered, limiting generality.
- Causal graphs are manually constructed and the evaluated queries are simple two-choice cause/effect pairs with unconditional d-separation.
- Human study is small (5 students, 100 queries) and the authors caution against direct comparison with LLMs.
Experimental setup
Datasets: DeScript COPA
Benchmarks: COLD (cake) COLD (shopping) COLD (train) COLD (tree) COLD (bus)
Metrics: accuracy (success rate)
Baselines: RoBERTa MNLI gpt-neo-125M gpt-neo-1.3B gpt-neo-2.7B gemma-2b phi-2 gpt-j-6B Llama-2-7b-chat-hf Mistral-7B-v0.1 gemma-7b Meta-Llama-3-8B
Builds on: COPA DeScript Mini Turing Test backdoor criterion d-separation
Closest papers by meaning
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? (NeurIPS 2024 · similarity 0.93)
- Language Agents Meet Causality -- Bridging LLMs and Causal World Models (ICLR 2025 · similarity 0.89)
- Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.89)
- Mitigating Hallucinations in Large Language Models via Causal Reasoning (AAAI 2026 · similarity 0.89)
- Nuance Matters: Probing Epistemic Consistency in Causal Reasoning (AAAI 2025 · similarity 0.89)
- Compositional Causal Reasoning Evaluation in Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.