Research map / Architectures and efficiency / RNNs and normalization
Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Maps sparse autoencoder features across consecutive layers and modules (residual, MLP, attention) via data-free decoder-weight cosine similarity, builds flow graphs, and uses them for multi-layer steering of Gemma 2 2B.
Problem
Sparse autoencoder features are interpretable within single layers, but how they emerge, persist, and transform across layers and modules of LLMs, and how to steer them jointly, is underexplored.
Approach
Match SAE features across consecutive layers and modules (residual, MLP, attention) using top-1 cosine similarity of decoder weights without data, compose these links into flow graphs, and steer multiple linked features across layers at once.
Key results
- Top-1 cosine matching achieves 65% deactivation success and 0.75 mean activation change, close to Top-1 Pearson (65%, 0.74) and below exhaustive search (73%, 0.83).
- Feature groups show distinct similarity-score distributions; early layers 0-5 show high uncertainty and 'From MLP' group diminishes after layer 18.
- Multi-layer (cumulative) steering improves deactivation of the 'Scientific concepts and entities' theme and reduces sensitivity to hyperparameters compared with single-feature interventions, at the cost of some coherence in activation tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Choose the top-1 cosine-similarity predecessor instead of a random feature from the top five matches | Feature deactivation · Feature deactivation | Successful deactivation rate | Helped |
| Use exhaustive search over active predecessor features instead of top-1 cosine matching | Feature deactivation · Feature deactivation, 1,894 features across two layers | Mean AC: 0.75 -> 0.83 | Helped |
| Use exhaustive search over active predecessor features instead of top-1 cosine matching | Feature deactivation · Feature deactivation, 1,894 features across two layers | Success rate: 65% -> 73% | Helped |
| Use top-1 cosine similarity instead of top-1 Pearson correlation for predecessor matching | Feature deactivation · Feature deactivation, 1,894 features across two layers | Mean AC: 0.74 -> 0.75 | No clear effect |
| Use cumulative interventions across layers instead of a single-layer intervention | Flow-graph model steering · Topic deactivation, low rescaling-coefficient regime | Deactivation score | Helped |
| Activate multiple related features cumulatively instead of steering one feature at a single layer | Flow-graph model steering · Topic activation | Coherence | Hurt |
| Use flow-graph-discovered features across layers instead of only the initial feature set | Flow-graph model steering · Topic deactivation | Deactivation score | Helped |
| Use Pearson correlation matching instead of top-1 cosine similarity matching | Feature predecessor identification · Gemma Scope; includes out-of-distribution Python code | Predecessor identification | Mixed |
Surprising findings
- Residual-stream predecessors are most influential for deactivation, while the MLP/attention predecessors matter less on their own.
- Attention-module SAE matches were almost absent for Gemma in the authors' own attention SAEs, plausibly reflecting SAE training rather than the model.
Limitations
- Cosine matching requires SAEs trained on aligned hidden-state spaces, so it cannot be applied across modules with different dimensions (e.g., Gemma Scope attention vs. residual SAEs).
- Own attention SAEs yield few attention-group matches, and the 'From nowhere' group is large, indicating uncovered causal dependencies.
- Steering evaluation uses a single prompt and few themes; hyperparameters were manually tuned.
Experimental setup
Datasets: FineWeb TinyStories AutoMathText PythonGithubCode
Metrics: successful deactivation rate activation change behavioral score coherence score Mann-Whitney U test p-value
Baselines: Top-1 Pearson correlation matching permutation matching single-layer steering random top-5 matching
Builds on: Gemma Scope JumpReLU Transcoders Llama Scope linear representation hypothesis
Closest papers by meaning
- LLM Layers Immediately Correct Each Other (NeurIPS 2025 · similarity 0.91)
- Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (ICLR 2025 · similarity 0.89)
- Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models (ICML 2024 · similarity 0.87)
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.87)
- Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders (NeurIPS 2025 · similarity 0.87)
- Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.