Research map / Architectures and efficiency / RNNs and normalization
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces sparse feature circuits, causal subgraphs over sparse autoencoder features discovered via linear attribution, and applies them to editing a classifier (SHIFT) and to fully unsupervised circuit discovery over thousands of behaviors.
Problem
Existing circuit analyses use polysemantic components (attention heads, neurons) and require researcher-specified hypotheses and curated data, so they cannot explain unanticipated model behaviors or support downstream editing.
Approach
Treat SAE feature activations and SAE error terms as nodes in the model's computation graph, use linear (attribution patching / integrated gradients) approximations of indirect effects to find causally implicated features and edges, and then ablate human-judged task-irrelevant features to remove spurious signals without disambiguating data.
Key results
- On Bias in Bios, SHIFT with retraining raises Pythia-70M worst-group accuracy from 24.4 to 89.0 and Gemma-2-2B from 18.2 to 92.9.
- Pythia-70M feature circuits: about 100 nodes explain the majority of performance, versus about 1500 neurons for half the performance.
- SHIFT without retraining matches or nears the feature skyline on Pythia (88.5 vs 88.5 profession accuracy), while neuron skyline reaches only 75.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ablate human-identified task-irrelevant SAE features | Pythia-70M classifier · Bias in Bios balanced set | Profession accuracy: 61.9 -> 88.5 | Helped |
| Ablate human-identified task-irrelevant SAE features | Pythia-70M classifier · Bias in Bios balanced set | Gender accuracy: 87.4 -> 54.0 | Helped |
| Ablate human-identified task-irrelevant SAE features | Pythia-70M classifier · Bias in Bios balanced set | Worst group accuracy: 24.4 -> 76.0 | Helped |
| Further fine-tune after Shift feature ablation (ablation) | Pythia-70M Shift classifier · Bias in Bios balanced set | Profession accuracy: 88.5 -> 93.1 | Helped |
| Further fine-tune after Shift feature ablation (ablation) | Pythia-70M Shift classifier · Bias in Bios balanced set | Gender accuracy: 54.0 -> 52.0 | Helped |
| Ablate human-identified task-irrelevant SAE features | Gemma-2-2B classifier · Bias in Bios balanced set | Profession accuracy: 67.7 -> 76.0 | Helped |
| Ablate human-identified task-irrelevant SAE features | Gemma-2-2B classifier · Bias in Bios balanced set | Gender accuracy: 81.9 -> 51.5 | Helped |
| Ablate human-identified task-irrelevant SAE features | Gemma-2-2B classifier · Bias in Bios balanced set | Worst group accuracy: 18.2 -> 50.0 | Helped |
| Further fine-tune after Shift feature ablation (ablation) | Gemma-2-2B Shift classifier · Bias in Bios balanced set | Profession accuracy: 76.0 -> 95.0 | Helped |
| Further fine-tune after Shift feature ablation (ablation) | Gemma-2-2B Shift classifier · Bias in Bios balanced set | Worst group accuracy: 50.0 -> 92.9 | Helped |
| Ablate randomly selected SAE features instead of human-selected features (ablation) | Shift classifier · Bias in Bios balanced set | Profession accuracy: null | No clear effect |
| Use neurons instead of SAE features for Shift (ablation) | Shift classifier · Bias in Bios balanced set | Worst group accuracy: Pythia-70M: 41.5; Gemma-2-2B: 5.6 | Hurt |
Surprising findings
- Ablating random SAE features has essentially no effect on probe performance, so SHIFT depends on careful feature selection.
- Circuits for agreement across a relative clause and across a prepositional phrase share over 85% of their features in Pythia and Gemma.
Limitations
- Success relies on existing or costly-to-train SAEs for the model; components not captured by SAEs remain uninterpretable.
- Much evaluation is qualitative; feature labeling is subjective and annotator-dependent.
- Gemma SHIFT without retraining damages performance, requiring retraining to restore accuracy.
Experimental setup
Datasets: The Pile Bias in Bios subject-verb agreement data adapted from prior work
Benchmarks: Bias in Bios balanced set subject-verb agreement held-out test split
Metrics: faithfulness completeness interpretability ratings from crowdworkers profession accuracy gender accuracy worst-group accuracy
Baselines: Neuron circuits Concept Bottleneck Probing Random feature ablations Neuron skyline Feature skyline Oracle
Builds on: attribution patching sparse autoencoders Gemma Scope integrated gradients concept bottleneck probing
Closest papers by meaning
- Transcoders find interpretable LLM feature circuits (NeurIPS 2024 · similarity 0.91)
- Analyze Feature Flow to Enhance Interpretation and Steering in Language Models (ICML 2025 · similarity 0.89)
- Scaling Sparse Feature Circuits For Studying In-Context Learning (ICML 2025 · similarity 0.88)
- Finding Interpretable Prompt-Specific Circuits in Language Models (NeurIPS 2026 · similarity 0.88)
- Explainable Neural Networks with Guarantee: A Sparse Estimation Approach (AAAI 2025 · similarity 0.88)
- Automatically Interpreting Millions of Features in Large Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.