Research map / Probabilistic and scientific modeling / Molecules and proteins
Optimal ablation for interpretability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes optimal ablation (OA), which replaces a model component with a learned constant minimizing expected loss, and shows it gives a canonical, lower-bias importance measure that improves circuit discovery, factual recall localization, and latent prediction in GPT-2 models.
Problem
Measuring component importance via ablation is inconsistent across methods (zero, mean, resample, counterfactual), and prior ablation choices can conflate true deletion effects with spoofing artifacts from out-of-distribution replacement values.
Approach
Replace a component's output with a constant a* that minimizes expected loss of the ablated model relative to the full model's predictions; this is proven to give the lowest ablation loss gap among all total (input-independent) ablation methods, and is used for circuit discovery (with a new UGS search), factual recall tracing, and a latent prediction lens (OCA lens).
Key results
- On IOI, optimal ablation's Delta is a median of 11.1% of zero-ablation Delta and 17.7% of resample Delta, with rank correlation 0.907 to counterfactual ablation.
- For factual recall at the last token, OAT yields AIE up to 42.6% for a 5-layer attention window vs 20.2% for Gaussian-noise tracing.
- OCA lens increases elicitation accuracy boost on DBPedia from 2.9% to 18.0% with true demonstrations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace components with an optimized constant rather than zero, mean, resampled, or counterfactual-mean values | GPT-2-small component importance on IOI · IOI | rank correlation with CF: 0.590; 0.825; 0.828; 0.833 -> 0.907 | Helped |
| replace components with an optimized constant rather than zero, mean, or resampled values | GPT-2-small component importance on IOI · IOI | median ablation loss gap: 11.1%; 33.0%; 17.7% -> 100% of the OA loss gap | Helped |
| use Uniform Gradient Sampling for circuit discovery | GPT-2-small circuit discovery · IOI | CF loss gap at 385 edges: 385 edges; 0.220 | Helped |
| use optimal ablation rather than counterfactual, mean, or resample ablation for excluded components | IOI circuits at 1,000 edges · IOI circuit discovery | loss gap: 32% lower than CF; 62% lower than mean; 88% lower than resample | Helped |
| use optimal ablation rather than Gaussian noise to remove subject information | GPT-2-XL causal tracing · Factual recall, attention layers 30–34 at last token | AIE: 20.2% -> 42.6% | Helped |
| use optimal ablation rather than Gaussian noise to remove subject information | GPT-2-XL causal tracing · Attention windows centered around layers 15–23 | average AIE: 7.0% -> 1.7% | Mixed |
| use OCA lens instead of tuned lens | GPT-2-XL latent prediction · GPT-2-XL prediction loss | predictive loss: below 0.01 around layer 35; tuned lens does not reach this point even at the last layer | Helped |
| use OCA lens instead of tuned lens | GPT-2-XL latent prediction · DBPedia, true demonstrations | elicitation accuracy boost: 2.9% -> 18.0% | Helped |
| use OCA lens instead of tuned lens | GPT-2-XL latent prediction · DBPedia, false demonstrations | elicitation accuracy boost: 19.2% -> 28.8% | Helped |
| use OCA lens instead of tuned lens | GPT-2-XL latent prediction · Causal faithfulness across layers | magnitude correlation and direction similarity | Helped |
Surprising findings
- Under optimal ablation, manual IOI and Greater-Than circuits appear approximately optimal for their size, unlike under other ablation methods which find much smaller circuits.
- OA-based circuits identified by UGS achieve up to 88% lower Delta than resample ablation at 1,000 edges on IOI.
Limitations
- Optimal ablation does not fully eliminate spoofing when a component carries strong exact information about the input.
- OA-discovered circuits may use mechanisms not used by the model on the subtask when multiple computational paths exist.
- Unclear whether factual recall localization helps downstream applications such as model editing.
Experimental setup
Datasets: IOI Greater-Than factual recall subject-attribute prompts (Meng et al.) 15 text classification datasets
Benchmarks: IOI Greater-Than factual recall (causal tracing) text classification elicitation accuracy
Metrics: ablation loss gap (KL divergence) AIE prediction loss causal faithfulness (magnitude correlation, direction simila rank correlation calibrated elicitation accuracy
Baselines: zero ablation mean ablation resample ablation counterfactual ablation ACDC edge attribution patching tuned lens Gaussian noise causal tracing
Builds on: activation patching ACDC edge attribution patching causal tracing tuned lens
Closest papers by meaning
- Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? (ICLR 2025 · similarity 0.85)
- Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference (AAAI 2025 · similarity 0.85)
- MIB: A Mechanistic Interpretability Benchmark (ICML 2025 · similarity 0.85)
- Local vs. Global Interpretability: A Computational Complexity Perspective (ICML 2024 · similarity 0.84)
- The Computational Complexity of Circuit Discovery for Inner Interpretability (ICLR 2025 · similarity 0.84)
- Finding Interpretable Prompt-Specific Circuits in Language Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.