Research map / Architectures and efficiency / RNNs and normalization
Observable Propagation: Uncovering Feature Vectors in Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces observable propagation (ObProp), a data-free method that backpropagates linear functionals (observables) on logits through transformer OV circuits and linearized nonlinearities to find feature vectors, plus a coupling coefficient for comparing them.
Problem
Finding linear feature vectors in transformers used for a task typically requires large labelled datasets and forward passes (probing, SAEs); the paper seeks data-light ways to find feature vectors and analyze how they are shared across tasks such as gender bias.
Approach
Treat task behavior as a linear functional (observable) on logits and propagate it backward through the model's OV circuits and first-order Taylor linearizations of MLPs and LayerNorms to obtain feature vectors without training or large datasets; define a coupling coefficient as a theoretically grounded estimate of proportionality between feature outputs.
Key results
- ObProp feature vector for subject pronouns reaches r^2 ≈ 0.945, matching logistic regression (trained) on the same task and exceeding mean difference (r^2 ≈ 0.899).
- On political party prediction, ObProp achieves r^2 ≈ 0.427 versus 0.295 for logistic regression trained on 3/4 of the dataset.
- The occupational-bias observable's feature vector has cosine similarity 0.966 with the subject-pronoun feature vector, suggesting shared gender features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ObProp instead of linear regression | Subject pronoun feature-vector extraction · Subject pronouns | r²: r^2 ≈ 0.945 vs. r^2 ≈ 0.945 | No clear effect |
| Use ObProp instead of the mean difference vector | Subject pronoun feature-vector extraction · Subject pronouns | r²: r^2 ≈ 0.899 -> r^2 ≈ 0.945 | Helped |
| Use ObProp instead of logistic regression | Political-party feature-vector extraction · Political parties | r²: r^2 ≈ 0.295 -> r^2 ≈ 0.427 | Helped |
| Use ObProp instead of the mean difference vector | Political-party feature-vector extraction · Political parties | r²: r^2 ≈ 0.0605 -> r^2 ≈ 0.427 | Helped |
| Use ObProp instead of logistic regression | C vs. Python feature-vector extraction · C vs. Python | AUC-ROC: AUC ≈0.9971 -> AUC ≈0.9974 | No clear effect |
| Use ObProp instead of the mean difference vector | C vs. Python feature-vector extraction · C vs. Python | AUC-ROC: AUC ≈0.9052 -> AUC ≈0.9974 | Helped |
| Ignore LayerNorm when computing feature-vector directions (ablation) | ObProp feature vectors · Feature vectors discussed in the paper | minimum cosine similarity to vectors computed through LayerNorm: 0.998 | No clear effect |
| Use attention-head feature-vector norms to identify important heads | ObProp · Subject and object pronoun prediction | path patching attribution | Mixed |
| Use the same feature vectors for gendered-pronoun and occupation prediction | GPT-Neo-1.3B · Gendered pronouns and occupational gender bias | cosine similarity: 0.966; 0.977 | Helped |
| Predict occupational-bias output from the dot product with the ObProp feature vector | GPT-Neo-1.3B · Occupational gender bias, 600 prompts | r²: 0.88 | Helped |
Surprising findings
- Head 18::11 has a large feature norm but little path-patching attribution, possibly due to QK-circuit effects.
- LayerNorm gradients have minimum cosine similarity 0.998 with LayerNorm-ignored feature vectors, so LayerNorms barely affect feature directions.
Limitations
- Primarily handles OV circuits and ignores QK-circuit computations such as induction heads.
- Relies on first-order linearization and an assumption of spherically symmetric hidden states for the coupling coefficient theorem.
- Empirical evaluation limited to a few tasks and small GPT models with hand-built artificial datasets.
Experimental setup
Datasets: The Pile Gender by Name dataset YouGov politician fame lists
Benchmarks: artificial gendered pronoun prompts political party prediction dataset C vs Python code snippets from The Pile
Metrics: r^2 AUC-ROC cosine similarity path patching attribution
Baselines: linear regression probe mean difference vector
Builds on: path patching linear probing logit lens-style OV circuit analysis
Closest papers by meaning
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.87)
- Understanding Transformers via N-Gram Statistics (NeurIPS 2024 · similarity 0.86)
- A distributional simplicity bias in the learning dynamics of transformers (NeurIPS 2024 · similarity 0.85)
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.85)
- Geometric Factual Recall in Transformers (NeurIPS 2026 · similarity 0.85)
- On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.