Research map / Large language models / Language, safety and interpretability
Automatically Interpreting Millions of Features in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an open-source automated pipeline (Delphi) that uses LLMs to generate natural-language interpretations for millions of SAE features and proposes five cheaper scoring methods, including intervention scoring.
Problem
SAEs trained on LLMs have millions of features, too many for humans to interpret manually, and existing simulation-based scoring of interpretations is expensive and focuses only on activating examples.
Approach
Treat interpretations as binary classifiers distinguishing activating from non-activating contexts and score them with cheaper methods (detection, fuzzing, surprisal, embedding), plus a new intervention scoring method that evaluates interpretability of a feature's causal effect on model output.
Key results
- Fuzzing has the highest Spearman correlation with human scores (0.69), versus simulation (0.60) and detection (0.59).
- Fuzzing and detection are at least 5x cheaper than simulation (AAO); embedding scoring costs about $50 per 100k features.
- Interpretations from the 131k-feature Gemma 2 9B SAE reach median fuzzing score 0.76 versus 0.51 for random interpretations; intervention scoring reveals features uninterpretable by correlational methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| collect activations using the Pile rather than RPJ-v2 (ablation) | Gemma 2 9B 131k residual SAE auto-interpretation · SAE feature interpretation evaluation | Detection: 0.74 (0.63--0.85) -> 0.76 (0.67--0.85) | No clear effect |
| collect activations using the Pile rather than RPJ-v2 (ablation) | Gemma 2 9B 131k residual SAE auto-interpretation · SAE feature interpretation evaluation | Fuzzing: 0.76 (0.67--0.86) -> 0.76 (0.67--0.86) | No clear effect |
| include token activation values in the explainer prompt (ablation) | Gemma 2 9B 131k residual SAE auto-interpretation · SAE feature interpretation evaluation | Fuzzing: 0.75 (0.65--0.86) -> 0.76 (0.67--0.86) | No clear effect |
| include token activation values in the explainer prompt (ablation) | Gemma 2 9B 131k residual SAE auto-interpretation · SAE feature interpretation evaluation | Detection: 0.73 (0.60--0.84) -> 0.74 (0.63--0.85) | No clear effect |
| add chain-of-thought reasoning when generating interpretations (ablation) | Llama 3.1 70B Instruct explainer · SAE feature interpretation evaluation | Fuzzing | No clear effect |
| show only top-activating examples instead of examples sampled across the whole distribution (ablation) | SAE feature interpretation generation · SAE feature interpretation evaluation | Fuzzing accuracy: 0.95 accuracy -> 0.8 accuracy; 0.57 accuracy -> 0.89 accuracy | Mixed |
| use fuzzing rather than simulation scoring | Auto-interpretation evaluation · Human-rated feature interpretations | Spearman correlation with human scores: 0.60 -> 0.69 | Helped |
| use detection rather than simulation scoring | Auto-interpretation evaluation · Human-rated feature interpretations | Spearman correlation with human scores: 0.60 -> 0.59 | No clear effect |
| use fuzzing scoring rather than simulation scoring | Auto-interpretation evaluation · 100 contexts per feature | Scoring cost with Llama 70B: 3.6k -> 676 | Helped |
| use detection scoring rather than simulation scoring | Auto-interpretation evaluation · 100 contexts per feature | Scoring cost with Llama 70B: 3.6k -> 588 | Helped |
Surprising findings
- Claude Sonnet 3.5 explanations were not much better than Llama 3.1 70B explanations despite being a stronger model.
- Surprisal scoring has the lowest correlation with the other scores and with simulation.
Limitations
- Explanations use 32-token contexts, so long-range features are not captured.
- Scoring methods have distinct failure modes and disagree with one another; surprisal has weak correlation with others.
- Interpretation length is not accounted for in current scores.
Experimental setup
Datasets: RedPajama-v2 10M token sample The Pile (uncopyrighted)
Benchmarks: Gemma Scope SAE feature explanations scored by fuzzing, detection, embedding, surprisal, simulation
Metrics: AUROC accuracy Pearson correlation Spearman correlation information value
Baselines: simulation scoring random interpretation randomly initialized TopK SAE neuron baselines
Builds on: simulation scoring (Bills et al. 2023) Patchscopes SelfIE Gemma Scope SAEs
Closest papers by meaning
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.92)
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.92)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.91)
- Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders (NeurIPS 2024 · similarity 0.90)
- I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders (AAAI 2026 · similarity 0.90)
- A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.