Research map / Label-efficient and robust learning / Recognition and long-tail
Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stress-tests the identifiability of mechanistic interpretability explanations on small Boolean-function MLPs, enumerating circuits, interpretations, algorithms and mappings to show many incompatible explanations coexist.
Problem
Mechanistic interpretability implicitly assumes a fixed behavior admits a unique explanation under its criteria (circuit error, consistent mapping, IIA); the paper asks whether these criteria actually guarantee unique explanations.
Approach
Borrowing identifiability from statistics, the paper defines identifiability of explanation and exhaustively enumerates Boolean circuits, logic-gate interpretations, algorithms and IIA-aligned mappings in tiny MLPs, showing non-identifiability at every stage of both where-then-what and what-then-where strategies.
Key results
- In the XOR example (2 hidden layers of size 3), 85 unique perfect circuits were found with an average of 535.8 consistent interpretations per circuit, and 159 perfect minimal mappings for two candidate algorithms.
- Median number of circuit interpretations grows from 38 to 910,000 as architecture width k goes from 2 to 5; median minimal mappings grow from 8 to 3,700 in the algorithm-first approach.
- Less than 2% of trained networks have exactly one valid minimal mapping, and no network has exactly one circuit interpretation.
- On a larger MLP trained on MNIST 0/1, the first sub-network alone admits 3,209 valid circuits, implying at least as many in the full network.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase architecture size from k=2 to k=5 | Boolean-gate MLP explanations · Boolean logic gates | Number of computational abstractions: 38 to 910,000; 8 to 3,700 | Helped |
| Train on more parallel target tasks, up to 4 tasks | Boolean-gate MLP explanations · Boolean logic gates, 1 to 6 parallel tasks | Number of interpretations: p=0.05 | Hurt |
| Add Gaussian noise N(0, 0.1) to training inputs (ablation) | What-then-where search · Boolean logic gates | Total mappings: 9.86 -> 9.89; p-value 0.959 | No clear effect |
| Add Gaussian noise N(0, 0.1) to training inputs (ablation) | What-then-where search · Boolean logic gates | Algorithms: 2.70 -> 2.56; p-value 0.766 | No clear effect |
| Add Gaussian noise N(0, 0.1) to training inputs (ablation) | Where-then-what search · Boolean logic gates | Circuits: 52.3 -> 28.6; p-value < 0.001 | Hurt |
| Add Gaussian noise N(0, 0.1) to training inputs (ablation) | Where-then-what search · Boolean logic gates | Interpretations per circuit (average): 2,270 -> 16,400; p-value < 0.001 | Helped |
| Add Gaussian noise N(0, 0.1) to training inputs (ablation) | Where-then-what search · Boolean logic gates | Total interpretations: 107,000 -> 285,000; p-value < 0.001 | Helped |
| Use a more unbalanced training distribution (ablation) | What-then-where search · Boolean logic gates | Algorithms: slope -0.21; p-value 0.857 | No clear effect |
| Use a more unbalanced training distribution (ablation) | What-then-where search · Boolean logic gates | Total mappings: slope -0.17; p-value 0.974 | No clear effect |
| Use a more unbalanced training distribution (ablation) | Where-then-what search · Boolean logic gates | Circuits: slope -326; p-value 0.001 | Hurt |
| Use a more unbalanced training distribution (ablation) | Where-then-what search · Boolean logic gates | Interpretations per circuit (average): slope 1,235; p-value 0.029 | Mixed |
| Use a more unbalanced training distribution (ablation) | Where-then-what search · Boolean logic gates | Total interpretations: slope -174,000; p-value 0.018 | Mixed |
Surprising findings
- Even randomly initialized networks that happen to implement a logic gate exhibit multiple computational abstractions.
- Increasing the number of training tasks reduces the number of interpretations only up to 4 tasks, after which the decrease is not significant.
Limitations
- Experiments restricted to toy MLPs and Boolean gates; generalization to transformers/LLMs not demonstrated.
- Circuit enumeration restricted to sparsity above 0.3 and two inputs/one output per gate, so counts are lower bounds.
- Interpretation search only considers logic gates and undercounts multi-neuron mappings.
Experimental setup
Datasets: MNIST (digits 0 and 1 subset) Boolean logic gate data (XOR, AND, OR, IMP, etc.)
Metrics: circuit error consistent mapping count intervention interchange accuracy (IIA) number of circuits, interpretations, algorithms and minimal
Builds on: interchange intervention accuracy (IIA) distributed alignment search causal abstraction circuit error / ACDC-style circuit discovery lottery ticket hypothesis
Closest papers by meaning
- Validating Mechanistic Interpretations: An Axiomatic Approach (ICML 2025 · similarity 0.91)
- Local vs. Global Interpretability: A Computational Complexity Perspective (ICML 2024 · similarity 0.88)
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability? (NeurIPS 2025 · similarity 0.88)
- The Computational Complexity of Circuit Discovery for Inner Interpretability (ICLR 2025 · similarity 0.88)
- Even-if Explanations: Formal Foundations, Priorities and Complexity (AAAI 2025 · similarity 0.88)
- From Neurons to Neutrons: A Case Study in Interpretability (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.