Research map / Large language models / AI and society
Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a causal framework that uses probabilities of necessity and sufficiency (PN/PS) computed from LLM-generated factual and counterfactual data to test whether LLMs reason, evaluating GPT-2, GPT-3.5-turbo and GPT-4 on math problems.
Problem
Assessing whether LLMs genuinely reason, particularly their counterfactual consistency, given that correct answers on direct questions may not reflect an understanding of underlying logical/causal structure.
Approach
Treats an LLM as an abstract machine (Hex framework) acting as a counterfactual data simulator, and compares LLM-estimated probabilities of causation (PN, PS) against ground-truth values computed from a causal reasoning graph, with bootstrap densities and overlap metrics.
Key results
- GPT-4 on Div6 gives PN=0.984 and PS=0.505 versus true PN=1 and PS=0.50 from the causal model.
- Only GPT-4 on the Div6 problem achieved overlap between estimated and true PN/PS densities; other model-problem results indicated weak reasoning.
- Perturbing true counterfactuals with probability 0.05 shifts estimated PN between 0.5 and 0.9, suggesting counterfactual error alone can overestimate reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4-generated factual and counterfactual datasets instead of the true datasets | Div6 PN estimate · Div6 | PN: 1 -> 0.984 | Mixed |
| Use GPT-4-generated factual and counterfactual datasets instead of the true datasets | Div6 PS estimate · Div6 | PS: 0.50 -> 0.505 | Mixed |
| Introduce random perturbations in the true counterfactuals with probability 0.05 | Div6 PN estimation · Div6 | PN: 0.5 and 0.9 | Hurt |
| Introduce random perturbations in the true counterfactuals with probability 0.2 | Div6 probability estimates · Div6 | PN and PS: 0.2 | Hurt |
| Use more complex models in the GPT family | GPT-2, GPT-3.5-turbo, and GPT-4 · Div6, EvenSum, and CandyParty | PN/PS alignment | Helped |
Surprising findings
- Counterfactual error rate of 0.2 yields negative, inconsistent PN/PS estimates.
- GPT-2 had accurate PN for EvenSum but notably less accurate PS estimates.
Limitations
- Requires access to causal reasoning graphs, limiting applicability where causal relations are hard to derive.
- Restricted to boolean variables.
- Results depend on the specific factual and counterfactual prompts used; prompts were not optimised.
Experimental setup
Datasets: synthetic math problems generated from causal reasoning graphs (Div6, EvenSum, CandyParty, ConPref)
Benchmarks: Div6 EvenSum CandyParty ConPref
Metrics: PN PS factual inconsistency rate (FIR) counterfactual inconsistency rate (CIR) gamma-PN-overlap gamma-PS-overlap
Baselines: GPT-2 GPT-3.5-turbo
Builds on: Hex framework probability of necessity and sufficiency (Tian and Pearl) structural causal models
Closest papers by meaning
- Reasoning over Uncertain Text by Generative Large Language Models (AAAI 2025 · similarity 0.90)
- Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning (NeurIPS 2025 · similarity 0.89)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.89)
- RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning (ICML 2025 · similarity 0.89)
- Are Language Models Efficient Reasoners? A Perspective from Logic Programming (NeurIPS 2025 · similarity 0.88)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.