Research map / Large language models / Language, safety and interpretability
Counterfactual reasoning: an analysis of in-context emergence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies in-context counterfactual reasoning in GPT-2-style Transformers and RNNs on a synthetic linear regression SCM requiring noise abduction, with mechanistic analysis (noise abduction heads, linear probes) and a preliminary extension to Lotka-Volterra SDE trajectories.
Problem
Whether language models can perform in-context counterfactual reasoning (noise abduction, intervention, prediction) rather than only observational or interventional prediction, and what architectural and data factors drive it.
Approach
Frames in-context counterfactual reasoning as a transformation on factual in-context observations, extends counterfactual identifiability to exchangeable (latent-theta mixture) data, and trains Transformers from scratch on synthetic sequences to test emergence, depth, attention, data diversity, and mechanistic signatures.
Key results
- Full GPT-2 Transformer reaches MSE of about 0.0025 on Lotka-Volterra counterfactual continuation, well below LSTM, GRU and Elman RNN (above 0.1).
- A linear probe on the residual stream after layer 2 of the 8-layer attention-only Transformer predicts the latent theta with adjusted R^2 of about 0.9104, staying above 0.9 in later layers.
- Diverse pre-training (higher effective support size) lowers in-context MSE for both in-distribution and OOD evaluation; a noise abduction head is found at layer 7 in a 12-layer, 8-head GPT-2 trained on E=1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove MLP sublayers (use attention-only) (ablation) | GPT-2 Transformer · Tanh counterfactual regression | MSE: 0.0132 vs 0.0058 | Hurt |
| remove MLP sublayers (use attention-only) (ablation) | GPT-2 Transformer · Sigmoid counterfactual regression | MSE: 0.0019 vs 0.0009 | Hurt |
| remove MLP sublayers (use attention-only) (ablation) | GPT-2 Transformer · Multiplicative-noise counterfactual regression | MSE: 0.0668 vs 0.0364 | Hurt |
| replace self-attention with MLP-only layers (ablation) | Transformer · Linear counterfactual regression | MSE | Hurt |
| increase Transformer depth while keeping 8 attention heads (ablation) | Full and attention-only Transformers · Linear counterfactual regression | MSE | Helped |
| increase pre-training data diversity (ablation) | Transformer · Linear counterfactual regression, in-distribution and OOD | MSE | Helped |
| use GPT-2 instead of Elman RNN | Autoregressive models · Linear counterfactual regression, contexts over 14 examples | MSE | Helped |
| use GPT-2 instead of LSTM or GRU | Autoregressive models · Linear counterfactual regression | MSE | No clear effect |
| use Full GPT-2 instead of recurrent models | Autoregressive models · Lotka-Volterra SDE, 20 time points | MSE: ≈ 0.0025 vs over 0.1 | Helped |
Surprising findings
- MLP-only models do not reason counterfactually and their loss stays at the theoretical variance regardless of context length.
- For the Full Transformer at E=5, the counterfactual reasoning behavior has not yet emerged within 50,000 training steps, and the phase transition occurs only later in training.
Limitations
- Experiments are on controlled synthetic data and small GPT-2-type models, not natural language.
- Assumes no observed confounders and only bivariate causal structure.
- Lotka-Volterra extension is described as preliminary and evaluated only on a single system.
Experimental setup
Datasets: synthetic linear regression SCM sequences (theta uniform in [-6,6]^5) Lotka-Volterra SDE trajectories
Benchmarks: synthetic in-context counterfactual regression (held-out sequences) Lotka-Volterra SDE counterfactual continuation
Metrics: mean squared error adjusted R^2 (linear probes) learning efficiency (number of in-context examples)
Baselines: LSTM GRU Elman RNN MLP-Only attention-only (AO) Transformer ordinary least squares estimator
Builds on: induction heads de Finetti theorem Bijective generation mechanisms identifiability (Nasr-Esfahany and Kiciman style GPT-2 AdamW
Closest papers by meaning
- What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (CVPR 2024 · similarity 0.91)
- DiCoFlex: Model-Agnostic Diverse Counterfactuals with Flexible Control (NeurIPS 2025 · similarity 0.89)
- Abstract Counterfactuals for Language Model Agents (NeurIPS 2025 · similarity 0.88)
- Reasoning Elicitation in Language Models via Counterfactual Feedback (ICLR 2025 · similarity 0.88)
- Gumbel Counterfactual Generation From Language Models (ICLR 2025 · similarity 0.87)
- Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.