Research map / Learning theory and algorithms / Combinatorial and search optimization
Validating Mechanistic Interpretations: An Axiomatic Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a set of abstraction-interpretation-inspired axioms (prefix/component equivalence and replaceability) for validating mechanistic interpretations, and applies them to a 2-layer Transformer trained on 2-SAT and to the known modular-addition model.
Problem
Mechanistic interpretations of neural networks are judged in an ad-hoc way; there is no formal criterion for whether a reverse-engineered program validly captures a network's computation compositionally.
Approach
Characterize a mechanistic interpretation as an approximate, compositional abstraction of a network via alpha (abstraction) and gamma (concretization) maps, with four statistically checkable axioms bounding per-prefix and per-component error in both intermediate representations and output behavior.
Key results
- The 2-SAT model reaches 99.76% test accuracy, and the first-block clause-parsing abstraction matches the model's intermediate states with epsilon about 0.0000374.
- For the second block, decision-tree neuron interpretations give prefix-equivalence epsilon about 0.182 versus about 0.309 for disjunction-only interpretations, quantifying the interpretability-fidelity trade-off.
- For modular addition, Nanda et al.'s interpretation satisfies the axioms with epsilon 0.000335 under rounding to three decimals, but rounding to one decimal gives epsilon 1 for the second component.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Enforce literal order consistency in the first-block clause representation | Clause-parser interpretation · 2-SAT test data, first block | epsilon (prefix equivalence): 0.0000374 -> 0.849 | Hurt |
| Use disjunction-only neuron interpretations instead of decision trees | 2-SAT interpretation · 2-SAT analysis test set, hidden layer | epsilon (prefix equivalence): 0.182 -> 0.309 | Hurt |
| Use disjunction-only neuron interpretations instead of decision trees | 2-SAT interpretation · 2-SAT analysis test set, hidden layer | epsilon (prefix replaceability): 0.0128 -> 0.00290 | Helped |
| Add hidden-neuron activation amplification in gamma2 (ablation) | Decision-tree neuron interpretation · 2-SAT analysis test set, hidden layer | epsilon (prefix replaceability): 0.249 -> 0.0128 | Helped |
| Add hidden-neuron activation amplification in gamma2 (ablation) | Disjunction-only neuron interpretation · 2-SAT analysis test set, hidden layer | epsilon (prefix replaceability): 0.135 -> 0.00290 | Helped |
| Use downstream-equivalence discretization instead of rounding to a single decimal place (ablation) | Modular-addition mechanistic interpretation · Modular addition (P=113), second component | epsilon: 1 -> 0.000335 | Helped |
Surprising findings
- Dropping the amplification step in the concretization map for the hidden layer raises prefix-replaceability error to about 0.249, showing sensitivity to how abstract states are mapped back.
- Discretization granularity strongly changes the verdict: the modular addition interpretation passes at three decimals but fails completely at one decimal.
Limitations
- The axioms are described as a minimal set that may not be sufficient for characterizing every mechanistic interpretation.
- The two extra axioms (strong and weak mechanistic derivability) are informal and not evaluated.
- Validation results depend on domain-specific discretization and on the choice of alpha/gamma maps, which are hand-designed or learned.
Experimental setup
Datasets: randomly generated 2-SAT formulas (10 clauses, up to 5 variables), labeled with Z3
Benchmarks: 2-SAT test set modular addition test set
Metrics: epsilon (Clopper-Pearson upper bound) for prefix/component e test accuracy
Baselines: causal abstraction causal scrubbing disjunction-only neuron interpretation
Builds on: abstract interpretation causal abstraction causal scrubbing Clopper-Pearson confidence intervals
Closest papers by meaning
- Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? (ICLR 2025 · similarity 0.91)
- Compact Proofs of Model Performance via Mechanistic Interpretability (NeurIPS 2024 · similarity 0.88)
- InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques (NeurIPS 2024 · similarity 0.86)
- Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models (ICML 2025 · similarity 0.86)
- From Neurons to Neutrons: A Case Study in Interpretability (ICML 2024 · similarity 0.86)
- The Computational Complexity of Circuit Discovery for Inner Interpretability (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.