Research map / Probabilistic and scientific modeling / Causal discovery
MIB: A Mechanistic Interpretability Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIB, a living benchmark with circuit localization and causal variable localization tracks, standardized tasks, counterfactual datasets, new faithfulness metrics (CPR/CMD), and leaderboards, evaluated on four models.
Problem
Mechanistic interpretability methods are compared ad hoc, with inconsistent metrics, tasks, and models, so it is unclear whether new methods truly improve on prior work.
Approach
A standardized benchmark with two tracks (circuit localization scored by area-based faithfulness metrics CPR and CMD plus AUROC on an InterpBench model; causal variable localization scored by interchange intervention accuracy) across fixed tasks, counterfactual datasets, and public/private test splits with leaderboards.
Key results
- EAP-IG-inputs with counterfactual ablations achieves the best average CMD and CPR across circuit localization tasks, with EAP-IG-activations and UGS competitive.
- In causal variable localization, DAS performs best across tasks (e.g., 88% IIA for answer token on ARC Easy with Gemma-2 and Llama-3.1).
- SAE features underperform neurons: DBM on SAE features is generally the worst method apart from full vector baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EAP-IG-inputs with counterfactual ablations rather than other circuit localization approaches | Circuit localization · MIB circuit localization tasks | CMD and CPR | Helped |
| Use edge-level attribution methods rather than node-level circuits | Circuit localization · MIB circuit localization tasks | CMD and CPR | Helped |
| Use counterfactual ablations instead of mean ablations | Circuit localization · IOI, GPT-2 | CMD: 0.29 -> 0.03 | Helped |
| Use counterfactual ablations instead of optimal ablations | Circuit localization · IOI, GPT-2 | CMD: 0.30 -> 0.03 | Helped |
| Use supervised DAS features instead of DBM-selected standard dimensions | Causal variable localization · MCQA, Gemma-2 answer token | IIA: 84 -> 95 | Helped |
| Use DBM-selected SAE features instead of DBM-selected standard dimensions | Causal variable localization · ARC (Easy), Gemma-2 answer token | IIA: 82 -> 70 | Hurt |
| Use DBM-selected PCA features instead of DBM-selected standard dimensions | Causal variable localization · MCQA, Gemma-2 answer token | IIA: 84 -> 57 | Hurt |
| Replace the selected features with the full untransformed hidden vector | Causal variable localization · ARC (Easy), Gemma-2 answer token | IIA: 82 -> 63 | Hurt |
| Use DBM-selected SAE features instead of DBM-selected standard dimensions | Causal variable localization · RAVEL, Gemma-2 continent | IIA: 66 -> 64 | No clear effect |
| Use supervised DAS rather than DBM for carry localization | Causal variable localization · Arithmetic (+), Llama-3.1 carry | IIA: 47 -> 54 | Mixed |
Surprising findings
- Edge activation patching, despite computing exact indirect effects, does not consistently outperform attribution methods.
- SAE features fail to beat standard neuron dimensions as causal-variable units of analysis.
Limitations
- No automated completeness metric for models other than the InterpBench model.
- Causal variable track assumes existence of the hypothesized high-level causal model; the Arithmetic carry-the-one baselines show weak results possibly because the variable is absent or nonlinearly encoded.
- Focuses solely on LLMs from a small set of four models and four tasks.
Experimental setup
Datasets: IOI Arithmetic MCQA ARC (Easy) ARC (Challenge) RAVEL InterpBench
Benchmarks: MIB circuit localization track MIB causal variable localization track InterpBench
Metrics: CPR CMD AUROC interchange intervention accuracy mean-squared error
Baselines: EAP EAP-IG EActP NAP NAP-IG IFR UGS Random PCA SAE Full Vector DBM
Builds on: InterpBench RAVEL Distributed Alignment Search Attribution Patching Desiderata-Based Masking
Closest papers by meaning
- Towards Best Practices of Activation Patching in Language Models: Metrics and Methods (ICLR 2024 · similarity 0.88)
- Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference (AAAI 2025 · similarity 0.88)
- Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (ICLR 2025 · similarity 0.87)
- Transcoders find interpretable LLM feature circuits (NeurIPS 2024 · similarity 0.86)
- Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models (ICML 2025 · similarity 0.86)
- Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.