Research map / Architectures and efficiency / RNNs and normalization

Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching

ICLR 2024 · Architectures and efficiency › RNNs and normalization · approach: Mechanistic circuit interpretability · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that 1-dimensional subspace activation patching can yield an illusory interpretability result by activating a dormant pathway through a causally disconnected MLP subspace, demonstrated on IOI in GPT-2 Small and factual recall in GPT2-XL, and linked to rank-1 (ROME) fact editing.

Problem

Subspace activation patching (e.g., via DAS) can make a model's output behave as if a feature changed, but the found subspace may not faithfully represent that feature; the paper asks whether such patches localize real features in the computation.

Approach

A patched subspace can combine a causally disconnected direction (in the kernel of the MLP down-projection) that varies with the feature and a dormant direction that is causal only when set out of distribution; patching then activates the dormant pathway. The paper links this to ROME rank-1 edits, which are approximately equivalent to 1-D subspace interventions.

Key results

• Patching the 1-D DAS direction in GPT-2 Small MLP8 gives FLDD 46.7%, dropping to 13.5% when its causally disconnected (ker W_out) component is removed and -8% for full MLP patch, while the residual-stream direction keeps about 124-141% FLDD across ablations.
• Rank-1 ROME edits in GPT2-XL layers 20-35 are closely approximated by 1-D subspace interventions, with cosine similarity near 1 in a range of layers.
• A 100-dimensional MLP8 subspace yields FLDD 62% and interchange accuracy 13%, while the residual-stream 100-D subspace yields FLDD 190% and interchange accuracy 89%.

Measured effects reported in the paper

Surprising findings

• Patching along a direction in an MLP whose causal effect is driven by a causally disconnected component, with the MLP as a whole not contributing to IOI, yields large apparent effects.
• Random MLP weights replacing MLP8 still yield a subspace that reduces logit difference even more, suggesting the illusion depends on layer position rather than MLP content.

Limitations

• Evidence for prevalence in practice is largely heuristic; only two settings (IOI and factual recall) are studied empirically.
• The illusion's magnitude is modest for single-component patches (interchange accuracy low), and ground truth for mechanistic explanation is conceptually hard to define.
• Fact editing evaluation uses only rewrite score and variance, without checking effects on related facts.

Experimental setup

Datasets: IOI synthetic templates CounterFact OpenWebText

Benchmarks: IOI task CounterFact

Metrics: fractional logit difference decrease interchange accuracy rewrite score cosine similarity total variance ratio

Baselines: full MLP patching full residual stream patching ROME

Builds on: distributed alignment search ROME activation patching indirect object identification circuit

Closest papers by meaning

• Towards Best Practices of Activation Patching in Language Models: Metrics and Methods (ICLR 2024 · similarity 0.87)
• The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching (NeurIPS 2026 · similarity 0.86)
• The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability? (NeurIPS 2025 · similarity 0.86)
• One SPACE to Rule Them All: Jointly Mitigating Factuality and Faithfulness Hallucinations in LLMs (NeurIPS 2025 · similarity 0.85)
• Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? (ICLR 2025 · similarity 0.84)
• DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions (ICML 2026 · similarity 0.84)