Research map / Learning theory and algorithms / Prediction and decision losses
The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that with unrestricted non-linear alignment maps, any neural network can be causally abstracted by any algorithm, making causal abstraction vacuous without assumptions on information encoding (the "non-linear representation dilemma").
Problem
Causal abstraction is used to claim that a neural network implements a high-level algorithm, but when alignment maps are unrestricted (non-linear), the notion may become trivially satisfied, so it is unclear whether it yields genuine interpretability.
Approach
Proves that under assumptions (countable inputs, input-injectivity and output-surjectivity of layers, matchable partial orderings, DNN solves the task), any algorithm is an input-restricted distributed abstraction of any DNN when alignment maps are unrestricted bijections; empirically uses reversible residual networks as non-linear alignment maps, including on randomly initialised models, to reach
Key results
- On randomly initialised Pythia language models, non-linear alignment maps reach 100% interchange intervention accuracy on the IOI task.
- On the hierarchical equality MLP, reversible-residual alignment maps achieve over 80% IIA even on randomly initialised networks.
- Linear (rotation) alignment maps show the layer-3 IIA degradation previously reported with DAS, which disappears with non-linear maps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a non-linear RevNet alignment instead of a linear alignment | DAS on the hierarchical equality task · Hierarchical equality, both equality relations and left equality relation | IIA | Helped |
| Use a non-linear RevNet alignment instead of identity or linear alignment | DAS with the identity of first argument algorithm · Hierarchical equality | IIA | No clear effect |
| Increase RevNet alignment complexity | DAS on the hierarchical equality task · Hierarchical equality, layer 1, random initialization | IIA | Helped |
| Train the MLP | DAS with RevNet alignment · Hierarchical equality, layer 1 | IIA | Helped |
| Use nonlinear RevNet alignment for randomly initialized models | Pythia models aligned to the ABAB-ABBA algorithm · IOI, Pythia model sizes | IIA: 100% | Helped |
| Train smaller Pythia models fully | RevNet alignment to the ABAB-ABBA algorithm · IOI, Pythia-31M and Pythia-70M | IIA | Hurt |
| Use an 8-layer RevNet instead of less complex alignments | DAS on Pythia-410M · IOI across training steps | IIA | Helped |
| Use a 4-layer RevNet instead of an 8-layer RevNet | DAS on Pythia-410M · IOI, training step 1000 | IIA | Hurt |
| Train the Pythia-410M model | DAS with 1- and 2-layer RevNet alignments · IOI | IIA | Helped |
| Use linear alignment instead of nonlinear RevNet alignment | DAS on Pythia-410M · IOI across training | IIA | Mixed |
| Evaluate with disjoint training and test name sets | Learned alignment maps · IOI | IIA | Hurt |
Surprising findings
- Randomly initialised language models can be perfectly aligned to the IOI algorithm with sufficiently complex alignment maps.
- Simpler alignment maps reach higher IIA as training progresses, and the identity-of-first-argument algorithm stays near 50% IIA for all map types.
Limitations
- The existence proof relies on a form of overfitting and gives no guarantees about learnability or generalisation of the alignment map.
- Input-injectivity and strict output-surjectivity assumptions may fail in practice (e.g., softmax bottleneck, floating point precision).
- Analysis is restricted to acyclic deterministic causal models.
Experimental setup
Datasets: hierarchical equality task data (synthetic) IOI dataset
Benchmarks: hierarchical equality indirect object identification
Metrics: interchange intervention accuracy
Baselines: identity alignment map (privileged bases) linear rotation alignment map (DAS)
Builds on: distributed alignment search causal abstraction pyvene
Closest papers by meaning
- Neural Causal Abstractions (AAAI 2024 · similarity 0.91)
- Causal Abstraction Inference under Lossy Representations (ICML 2025 · similarity 0.90)
- Learning Linear Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity (NeurIPS 2024 · similarity 0.89)
- Causal Abstraction Learning based on the Semantic Embedding Principle (ICML 2025 · similarity 0.89)
- Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? (ICLR 2025 · similarity 0.88)
- How are linear representations learned? Exact solutions to the dynamics of abstraction (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.