Research map / Large language models / Language, safety and interpretability
Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies how corruption method (Gaussian noise vs symmetric token replacement), evaluation metric (probability vs logit difference vs KL), and sliding window patching affect activation patching localization results, and gives best-practice recommendations.
Problem
Activation patching is a standard localization tool in mechanistic interpretability, but papers vary in corruption methods, metrics, and window choices with no consensus, so reported localization and circuit results may be sensitive to these hyperparameters.
Approach
A first systematic study of activation patching's degrees of freedom: compares Gaussian noise vs symmetric token replacement corruption, probability vs logit difference metrics, and sliding window vs single-layer summation, showing disparate localization outcomes and arguing GN induces off-distribution internals and probability can miss negative components.
Key results
- On GPT-2 XL factual recall, GN yields a peak 2x-5x higher than STR across window sizes, but STR gives stronger concentration for addition/subtraction in GPT-J.
- On IOI with STR corruption, probability fails to detect negative Name Mover 11.10 since corrupted-run IO probability is only 0.03, while logit difference detects it.
- Corrupting S1 and IO instead of S2 lets most metric/corruption combinations recover all three Name Mover heads (9.6, 9.9, 10.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STR instead of GN corruption | Activation patching for factual recall in GPT-2 XL · GPT-2 XL factual recall | peak MLP patching effect: GN peak is 2×--5× higher than STR; STR is stronger by up to 4× on GPT-J addition and subtraction | Mixed |
| Use STR instead of GN corruption | Activation patching for IOI circuit discovery · IOI | attention-head localization | Mixed |
| Use STR instead of GN corruption | Activation patching for the greater-than circuit · Greater-than | attention-head localization | Helped |
| Use probability instead of logit difference (ablation) | Activation patching across subject-token positions · GPT-2 XL factual recall, STR, window size 5 | last-to-middle subject-token effect ratio: 4.33× vs 1.22× | Mixed |
| Use probability instead of logit difference (ablation) | Activation patching across subject-token positions · GPT-2 XL factual recall, GN, window size 5 | last-to-middle subject-token effect ratio: 1.74× vs 0.77× | Mixed |
| Use probability as the metric (ablation) | IOI negative-head detection under STR · IOI | detection of negative head 11.10: mean 0.003, SD 0.015, and 11.10 receives -0.022 | Hurt |
| Use probability instead of logit difference (ablation) | IOI negative-head detection with random-name corruption · IOI | detection of negative Name Movers: 5\mathrm{e}{-4} | Hurt |
| Jointly patch adjacent MLP layers rather than sum individual-layer effects (ablation) | Activation patching for factual recall · GPT-2 XL factual recall | peak patching effect: 1.40×, 1.75× and 1.59× | Helped |
| Corrupt S1 and IO instead of S2 (ablation) | Activation patching for IOI circuit discovery · IOI | Name Mover head localization | Helped |
| Restore the values of S-Inhibition heads under GN rather than STR (ablation) | IOI Name Mover behavior · IOI | logit difference: 1.04 -> 0.49 | Hurt |
| Use the original-IO probability as the metric under strong random-name corruption (ablation) | Activation patching for IOI negative-head discovery · IOI | negative Name Mover detection: 5\mathrm{e}{-4} | Hurt |
Surprising findings
- GN corruption makes head 0.10 appear as a negative contributor to IOI performance, which disappears under STR and conflicts with prior evidence that it helps.
- Sliding window patching amplifies weak single-layer localization peaks, with 1.40x-1.75x higher peak than summing single-layer effects.
Limitations
- Experiments limited to decoder-only language models up to 6B parameters
- Only overriding corrupted activations by clean ones; patching in the reverse direction not studied
- Tentative evidence for OOD behavior under Gaussian noise corruption
Experimental setup
Datasets: CounterFact Known1000 IOI prompts (Wang et al.) PairedFacts (constructed)
Benchmarks: factual recall (CounterFact/Known1000) IOI circuit greater-than Python docstring basic arithmetic
Metrics: probability logit difference KL divergence patching effect standard deviation detection threshold
Baselines: Gaussian noise corruption probability metric ROME-style causal tracing
Builds on: causal tracing (ROME) IOI circuit analysis path patching TransformerLens
Closest papers by meaning
- MIB: A Mechanistic Interpretability Benchmark (ICML 2025 · similarity 0.88)
- Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching (ICLR 2024 · similarity 0.87)
- Steering When Necessary: Flexible Steering Large Language Models with Backtracking (NeurIPS 2025 · similarity 0.87)
- Exploring Activation Patterns of Parameters in Language Models (AAAI 2025 · similarity 0.87)
- Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models (ICML 2024 · similarity 0.87)
- Joint Localization and Activation Editing for Low-Resource Fine-Tuning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.