Research map / Probabilistic and scientific modeling / Causal discovery
Gumbel Counterfactual Generation From Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates language models as well-founded structural equation models via the Gumbel-max trick and uses hindsight Gumbel sampling to generate true string counterfactuals, then shows common intervention methods cause unintended side effects.
Problem
Interventions on language models (knowledge editing, linear steering, instruction tuning) cannot answer counterfactual questions about how a specific observed string would have been generated under the modified model, and prior counterfactual analyses lacked a precise causal formalization for LMs.
Approach
Frame an LM as a well-founded SEM where next-token choices are argmax of logits plus i.i.d. Gumbel noise; infer the posterior of the exogenous noise given an observed string (hindsight Gumbel sampling) and rerun the intervened model with that noise to obtain the counterfactual string.
Key results
- MEMIT yields the most precise intervention, with median normalized edit distance of about 10-15% for the Louvre and Koalas edits.
- On Louvre-focused sentences, 54.6% of counterfactuals mention Rome as the Louvre's location while 45.4% still mention Paris.
- Intervention methods (steering, instruction tuning, knowledge editing) produce counterfactuals with unintended semantic drift on unrelated Wikipedia prompts, showing side effects.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MEMIT to edit the Louvre location from Paris to Rome | GPT2-XL · Wikipedia prompts, Louvre edit | normalized edit distance: 10-15% | Mixed |
| Apply MEMIT to edit the Koalas' habitat from Australia to New Zealand | GPT2-XL · Wikipedia prompts, Koalas edit | normalized edit distance: 10-15% | Mixed |
| Edit the Louvre location from Paris to Rome | GPT2-XL · Louvre-focused counterfactuals | counterfactuals mentioning Rome: 54.6% | Helped |
| Edit the Louvre location from Paris to Rome | GPT2-XL · Louvre-focused counterfactuals | counterfactuals still mentioning Paris: 45.4% | Hurt |
| Apply common model interventions, including knowledge editing and linear steering | Language models · Wikipedia prompts | semantic similarity of original and counterfactual text | Hurt |
Surprising findings
- Even minimal interventions such as steering vectors that modify a tiny fraction of parameters cause considerable semantic drift in unrelated continuations.
- Counterfactuals for Louvre-focused sentences are not minimal, changing unrelated parts of the sentence as well.
Limitations
- MEMIT replication on LLaMA3-8b failed, so knowledge-editing analysis is limited to GPT2-XL.
- Gumbel-max is one of several counterfactually stable choices; identifiability holds only under the Thurstone assumption.
- Only a 25-token continuation length is evaluated and results rely on a small number of sentences (75 Louvre sentences).
Experimental setup
Datasets: English Wikipedia sentences (prompts) Bios dataset
Benchmarks: Wikipedia continuation prompts (500 per model pair) Louvre-focused GPT2-XL sentences (75)
Metrics: normalized edit distance fraction of counterfactuals mentioning Rome
Baselines: MEMIT Honest LLaMA MiMiC LLaMA3-8b-Instruct
Builds on: Gumbel-max trick structural equation models MEMIT MiMiC Inference-time intervention (Honest LLaMA)
Closest papers by meaning
- Counterfactual reasoning: an analysis of in-context emergence (NeurIPS 2025 · similarity 0.87)
- Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals (ICLR 2024 · similarity 0.86)
- Counterfactual Generative Modeling with Variational Causal Inference (ICLR 2025 · similarity 0.86)
- Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors (ICML 2025 · similarity 0.85)
- Towards Characterizing Domain Counterfactuals for Invertible Latent Causal Models (ICLR 2024 · similarity 0.85)
- Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.