Research map / Architectures and efficiency / RNNs and normalization
What Does a Sparse Autoencoder Feature Do? A Weight-Based Account
Yiting Liu, Zhihong Deng
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a weight-based account of sparse autoencoder features that characterizes the computational roles features inherit from the SAE training objective, rather than inferring semantics only from activation patterns.
Problem
Current SAE interpretation pipelines infer feature semantics from activation patterns, implicitly assuming every feature has a semantic explanation and overlooking computational roles inherited from the SAE training objective.
Approach
Interprets SAE features through their encoder/decoder weights rather than activation patterns, arguing some features may serve computational roles set by the training objective instead of carrying semantic meaning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Zero out a weight-identified semantic feature (do(z=0)) vs matched-norm non-semantic feature (ablation) | Gemma-2-9B and Llama-3.1-8B SAE features · next-token logit at top-1 token | median |delta logit|: 0.8-7.1 vs 0.02-0.13 | Improved |
| Zero out weight-identified query specialists / key hubs vs matched baselines (ablation) | SAE features · next-layer attention | effect on target attention quantity: 43x-2161x (queries), 9x-37x (keys) | Improved |
| Use untied embeddings (Llama) instead of tied (Gemma) | SAE semantic depth profile · Llama-3.1-8B vs Gemma-2 | pass rate across depth | Mixed |
| Ablate any single of the three semantic metrics (ablation) | three-metric classifier · Gemma-2-2B/9B | joint pass rate | Mixed |
| Vary calibration percentile p in [40,60] (ablation) | classifier · six model-projection combinations | Spearman with p=50 profile: minimum 0.878 | No clear effect |
| Apply per-feature normalization vs unnormalized baseline (ablation) | attention-participation scores · Gemma-2-9B | Spearman rank correlation: 0.902-0.985 (queries), 0.876-0.980 (keys) | No clear effect |
| Compare semantic vs non-semantic features on attention weight | SAE features · Gemma-2 query-side attention | attention weight | Hurt |
| Compare semantic vs non-semantic features in Llama input phase | SAE features · Llama-3.1-8B layers before 11 | attention weight | Improved |
| Use Gemma-2 instead of Llama for attention specialist counts | SAE features · specialist populations by depth | 2-layer oscillation | Mixed |
Experimental setup
Builds on: sparse autoencoders
Closest papers by meaning
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.92)
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.92)
- Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression (NeurIPS 2026 · similarity 0.92)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.92)
- Automatically Interpreting Millions of Features in Large Language Models (ICML 2025 · similarity 0.92)
- A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
