Research map / Architectures and efficiency / Attention and transformers
InversionView: A General-Purpose Method for Reading Information from Neural Activations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InversionView, which trains a conditional decoder to sample inputs whose activations are close to a query activation, revealing the information content of activations in small transformers and GPT-2 circuits.
Problem
Existing interpretability methods (probes, vocabulary projection, causal interventions) do not directly reveal the human-readable information encoded in neural activations of language and sequence models.
Approach
Define the information in an activation as its epsilon-preimage (the set of inputs producing similar activations) and approximate it by training an autoregressive decoder conditioned on cached activations to sample inputs; distances are then checked on the probed model.
Key results
- Character counting probed model reaches 99.53% test accuracy; decoder in-preimage rate averages 67.7% across activation sites.
- 3-digit addition probed model reaches 98.01% test accuracy; the InversionView-derived circuit is causally verified by activation patching.
- On GPT-2 small IOI, InversionView agrees with prior interpretations for 23 of 26 heads identified by Wang et al.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compare layer 0 activations before and after the MLP | Character-counting transformer · Character counting | prominence of target-character count in activation geometry | Helped |
| Use the layer 1 colon residual stream to carry count information | Character-counting transformer · Character counting | count information represented at the colon | Helped |
| Patch a^{0,0}_{tc} or a^{1,0}_{:} from a contrast example | Character-counting transformer · Character counting | logit-difference effect | Helped |
| Decode activations with InversionView | GPT-2 Small IOI circuit · Indirect object identification | agreement with previously inferred head information: 3 of 26 | Helped |
| Use InversionView to inspect addition activations and trace the A1 circuit | 3-digit addition transformer · 3-digit addition, A1 | identification of the cause of model errors | Helped |
| Increase sampling temperature from 1.0 to 2.0 (ablation) | InversionView decoder · 3-digit addition | sampling-probability smoothness within the epsilon-preimage | Helped |
| Add noise with coefficient eta=0.1 to the query activation (ablation) | InversionView decoder · 3-digit addition | sampling-probability smoothness within the epsilon-preimage | Helped |
| Replace activation conditioning in self-attention with encoder-decoder attention (ablation) | InversionView decoder · Decoder architecture | decoder performance | No clear effect |
| Vary the epsilon threshold (ablation) | InversionView · Character counting and 3-digit addition | granularity of inferred activation information | Mixed |
| Patch x^{0,postStream}_{+} with its test-set mean (ablation) | 3-digit addition transformer · 3-digit addition | effect on model output: $8.1\times10^{-8}$; $5.4\times10^{-7}$; $2.5\times10^{-8}$; $3.6\times10^{-9}$ KL divergence | No clear effect |
Surprising findings
- For the tens-place digits in addition, the model encodes digit pairs only by their sum when F2+S2=9, a special cascading-carry case.
- The claimed circuit deficiency (missing ones-to-hundreds carry) accounts for all of the model's mistakes on addition.
Limitations
- Relies on a black-box decoder whose completeness must be validated via counterexamples; interpretation is still manual and laborious.
- Only models up to 1.5B parameters are studied; factual recall is preliminary.
- Distance metric choice (Euclidean vs cosine) may be suboptimal given non-isotropic representation geometry.
Experimental setup
Datasets: synthetic character counting inputs IOI prompts 3-digit addition strings
Benchmarks: character counting test set 3-digit addition test set IOI circuit heads from Wang et al.
Metrics: test accuracy in-preimage rate logit difference under activation patching KL divergence logit decrement rate
Baselines: supervised probing vocabulary projection activation patching
Builds on: activation patching path patching Nanda-style addition circuit analysis IOI circuit (Wang et al.)
Closest papers by meaning
- Network Inversion of Convolutional Neural Nets (Student Abstract) (AAAI 2025 · similarity 0.84)
- Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants (NeurIPS 2026 · similarity 0.83)
- Since Faithfulness Fails: The Performance Limits of Neural Causal Discovery (ICML 2025 · similarity 0.82)
- Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations (NeurIPS 2025 · similarity 0.82)
- Linear Explanations for Individual Neurons (ICML 2024 · similarity 0.82)
- Tilt your Head: Activating the Hidden Spatial-Invariance of Classifiers (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.