Research map / Probabilistic and scientific modeling / Causal discovery

The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching

NeurIPS 2026 · Probabilistic and scientific modeling › Causal discovery · approach: Causal effect identification · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that activation patching's noising NIE equals PIE plus a mediated interaction term with the residual bypass, derives its second-order form and combinatorial multi-component decomposition, and uses GPT-2 IOI to argue INT is a diagnostic rather than a nuisance.

Problem

Activation patching is assumed to estimate the causal (path-specific) effect of a component, but the noising NIE includes interaction terms with the residual bypass, so component rankings and faithfulness scores may be distorted.

Approach

Re-derives the activation patching estimand from causal mediation analysis, showing noising NIE = PIE + INT where INT is the mediated interaction with the bypass; proves INT is a second-order Hessian term that vanishes for locally affine maps, and decomposes multi-component patching into PIE, individual and cross-component interactions.

Key results

• Rank correlation between mean NIE and PIE across heads drops to rho=0.509 for IOI with pABC corruption and 0.517 for Greater Than, versus 0.989 for SVA.
• In GPT-2 IOI, L9H9 has PIE=+2.575 but INT=-2.338, so NIE=+0.237 (rank 13), illustrating backup compensation hidden by NIE.
• Under pABC mean ablation, summed PIEs are +5.775 logit-diff while the jointly measured faithfulness is only 1.523, with cross-interaction -4.045 accounting for most of the gap.

Surprising findings

• Under symmetric token-replacement corruption INT is suppressed and NIE rankings reduce to PIE rankings, so the pABC counterfactual is what makes context-specific heads visible.
• Tracr-compiled piecewise-affine models have zero INT, so common interpretability benchmarks may not reflect INT conditions in real transformers.

Limitations

• Exact patching over all heads only on small models (GPT-2 small, Pythia-70M, Qwen2.5-0.5B); scalability to larger models unclear.
• Interpretation of INT signs as functional roles relies on the IOI circuit's pre-registered functional labels and may not transfer to diffuse functionality.
• Second-order approximation holds only locally; combinatorial decomposition requires exponentially many patches to fully resolve.

Experimental setup

Datasets: IOI dataset (N=1000 prompt pairs) SVA Gender Bias Greater Than

Benchmarks: IOI SVA Gender Bias Greater Than

Metrics: Spearman rank correlation NIE PIE INT faithfulness difference logit difference

Baselines: NIE ranking PIE ranking

Builds on: natural indirect effect path-specific effect activation patching IOI circuit (Wang et al.)

Closest papers by meaning

• Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching (ICLR 2024 · similarity 0.86)
• Towards Best Practices of Activation Patching in Language Models: Metrics and Methods (ICLR 2024 · similarity 0.85)
• Higher-Order Causal Message Passing for Experimentation with Complex Interference (NeurIPS 2024 · similarity 0.85)
• Identifying General Mechanism Shifts in Linear Causal Representations (NeurIPS 2024 · similarity 0.84)
• Detecting and Measuring Confounding Using Causal Mechanism Shifts (NeurIPS 2024 · similarity 0.84)
• Identifiability Guarantees for Causal Disentanglement from Purely Observational Data (NeurIPS 2024 · similarity 0.84)