Research map / Architectures and efficiency / RNNs and normalization
InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InterpBench, a benchmark of semi-synthetic transformers with known circuits, trained via Strict Interchange Intervention Training (SIIT), and uses it to compare circuit discovery methods such as ACDC, subnetwork probing, and EAP.
Problem
Mechanistic interpretability circuit discovery methods cannot be validated because the true circuits of natural neural networks are unknown; Tracr-compiled transformers with known circuits are unrealistic (sparse, no superposition).
Approach
Extends Interchange Intervention Training with a strictness loss that intervenes on low-level nodes not aligned to the high-level circuit and penalizes changes to the output, so that non-circuit components cannot affect the output; this yields realistic transformers with verifiably known circuits.
Key results
- SIIT models show low node effect for non-circuit nodes while IIT models do not, matching Tracr behavior on the main 16 tasks.
- ACDC significantly outperforms Node Subnetwork Probing (p-value about 0.0004, A12 about 0.742) on InterpBench.
- EAP with integrated gradients is statistically indistinguishable from ACDC (p-value 0.099), while regular EAP performs poorly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add strictness loss by intervening on non-aligned low-level nodes | IIT-trained transformers · InterpBench Tracr tasks | non-circuit node effect on output | Helped |
| Use IIT without the strictness loss (Weight_SIIT set to 0) (ablation) | SIIT training · InterpBench Tracr tasks | non-circuit node effect on output | Hurt |
| Use SIIT rather than Tracr-generated models | Circuit discovery realism evaluation · InterpBench Tracr tasks | correlation with naturally trained model accuracy | Helped |
| Use ACDC rather than Node SP | Circuit discovery on SIIT transformers · InterpBench Tracr tasks | edge ROC AUC: p-value ≈ 0.0004; Â12 ≈ 0.742 | Helped |
| Use ACDC rather than Edgewise SP | Circuit discovery on SIIT transformers · InterpBench Tracr tasks | edge ROC AUC: 0.028417 | Helped |
| Use ACDC rather than EAP | Circuit discovery on SIIT transformers · InterpBench Tracr tasks | edge ROC AUC: 0.000061 | Helped |
| Use EAP-IG instead of regular EAP | Circuit discovery on SIIT transformers · InterpBench Tracr tasks | edge ROC AUC: 0.000648 | Helped |
| Set Weight_SIIT to 0.5, with Weight_IIT and Weight_behavior set to 1 (ablation) | SIIT · Four randomly selected InterpBench cases | accuracy, IIA, and SIIA: 0.5 | Helped |
| Set Weight_SIIT below 0.5 (ablation) | SIIT · Four randomly selected InterpBench cases | SIIA: up to 20% | Hurt |
| Set Weight_SIIT above 0.5 (ablation) | SIIT · Four randomly selected InterpBench cases | IIA: between 0% and 10% | Hurt |
| Mean-ablating all nodes outside the ground-truth circuit (ablation) | SIIT models · InterpBench Tracr tasks | accuracy | Mixed |
| Zero-ablating nodes outside the ground-truth circuit (ablation) | SIIT models · InterpBench Tracr tasks | non-circuit node effect on output | Hurt |
Surprising findings
- Regular EAP performs poorly relative to ACDC on SIIT models, plausibly because SIIT models have very small gradients.
- Non-circuit nodes in GPT-2 small's IOI circuit can have higher resample-ablation effects than some circuit nodes.
Limitations
- Models are very small with only one algorithmic circuit each, so results may not transfer to large LLMs.
- Sparse autoencoders were not evaluated because of the small number of features.
- Zero ablations reveal non-circuit nodes with significant effects, suggesting circuits may not be fully faithful.
Experimental setup
Datasets: Tracr-compiled RASP programs TracrBench circuits IOI task dataset
Benchmarks: InterpBench
Metrics: node effect normalized KL divergence AUROC Wilcoxon-Mann-Whitney p-value Vargha-Delaney A12 correlation coefficient
Baselines: IIT Tracr-generated transformers ACDC Subnetwork Probing EAP EAP-IG
Builds on: Interchange Intervention Training Tracr Automatic Circuit DisCovery Subnetwork Probing Edge Attribution Patching
Closest papers by meaning
- Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? (ICLR 2025 · similarity 0.87)
- Transcoders find interpretable LLM feature circuits (NeurIPS 2024 · similarity 0.86)
- Validating Mechanistic Interpretations: An Axiomatic Approach (ICML 2025 · similarity 0.86)
- Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition (ICLR 2025 · similarity 0.86)
- The Computational Complexity of Circuit Discovery for Inner Interpretability (ICLR 2025 · similarity 0.86)
- MIB: A Mechanistic Interpretability Benchmark (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.