Research map / Architectures and efficiency / Attention and transformers
Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes attention heads and MLP layers of GPT-2 Small into orthogonal singular directions via SVD on augmented weight matrices, learns sparse direction-level masks, and interprets IOI head 9.6 and gender-pronoun "logit receptor" directions.
Problem
Mechanistic interpretability typically treats attention heads and MLPs as atomic units, missing functional substructure that may be multiplexed within a single component.
Approach
Model each attention/MLP matrix (with bias folded in as an augmented row) by its SVD and learn a sparse diagonal mask over singular directions using a KL-plus-L1 objective with clean/corrupted joint inputs, so that task-relevant directions within a head or MLP can be identified and interpreted.
Key results
- On IOI, learned masks keep about 9% of singular directions (relative sparsity 91.32%) with KL 0.21 and exact match 0.77.
- On GT and GP, over 95% of directions can be pruned (relative sparsity 95.21% and 96.81%) with KL 0.23 and 0.13 respectively.
- Scalar swap interventions on gender-related OV directions flip GPT-2 pronoun predictions to the opposite gender in up to 100% of cases at amplification 5 or higher.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse learned singular-direction masks instead of the full component | GPT-2 Small · IOI | accuracy: 0.79$\pm$0.05 -> 0.70$\pm$0.07 | Hurt |
| Use sparse learned singular-direction masks instead of the full component | GPT-2 Small · GP | accuracy: 0.77$\pm$0.04 -> 0.75$\pm$0.04 | Hurt |
| Swap all gender-associated direction activations to opposite-gender means and scale singular values by 1.0 (ablation) | GPT-2 Small · GP, masculine prompts | pronoun logit difference: $+2.53 \pm 1.48$ -> $-1.10 \pm 0.83$ | Hurt |
| Swap all gender-associated direction activations to opposite-gender means and scale singular values by 20.0 (ablation) | GPT-2 Small · GP, masculine prompts | pronoun logit difference: $+2.53 \pm 1.48$ -> $-42.31 \pm 8.57$ | Hurt |
| Swap all gender-associated direction activations to opposite-gender means and scale singular values by 20.0 (ablation) | GPT-2 Small · GP, feminine prompts | pronoun logit difference: $+2.84 \pm 2.12$ -> $-40.82 \pm 15.44$ | Hurt |
Surprising findings
- The top singular direction of the gender-related OV matrix (L11.H7.SV0, sigma 22.07) receives near-zero mask because its activation does not differ between he and she contexts.
- The first token in a sequence receives activations 20-25x larger than subsequent tokens on the top singular direction of several heads.
Limitations
- Analysis treats each augmented matrix independently, missing cross-component interactions.
- Assumes singular directions are interpretable and causally relevant without formal justification; masks restricted to fixed axis-aligned singular vectors.
- Evaluated only on GPT-2 Small and standard synthetic benchmarks, not larger or instruction-tuned models.
Experimental setup
Datasets: IOI Greater-Than Gender Pronoun
Benchmarks: IOI Greater-Than Gender Pronoun
Metrics: KL divergence exact match accuracy sparsity (relative and full) logit difference flip rate
Baselines: ACDC top-k magnitude SVD masking random SVD masking
Builds on: transformer circuits framework ACDC TransformerLens
Closest papers by meaning
- Circuit Component Reuse Across Tasks in Transformer Language Models (ICLR 2024 · similarity 0.91)
- Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition (ICLR 2025 · similarity 0.88)
- Characterizing the Expressivity of Fixed-Precision Transformer Language Models (NeurIPS 2025 · similarity 0.88)
- Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024 · similarity 0.87)
- Transformer Layers as Painters (AAAI 2025 · similarity 0.87)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.