Research map / Architectures and efficiency / RNNs and normalization
Dense SAE Latents Are Features, Not Bugs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper studies dense SAE latents (active on >10% of tokens), showing they reflect intrinsic residual-stream subspaces, form antipodal pairs, and fall into a taxonomy of functional classes across layers of Gemma 2 2B.
Problem
Dense SAE latents that activate on a large fraction of tokens are often dismissed as training artifacts; it is unclear whether they reflect meaningful model computation.
Approach
Ablating the dense-latent subspace before retraining an SAE eliminates new dense latents, indicating dense latents track intrinsic residual-stream directions; dense latents pair antipodally, and a taxonomy (position, context-binding, nullspace/entropy, alphabet, part-of-speech, PCA) links them to functional roles.
Key results
- Retraining an SAE after ablating the dense-latent subspace yields far fewer high-density latents than baseline or sparse-subspace ablation on Gemma 2 2B.
- Dense latents with density above 0.3 mostly have antipodality scores above 0.9.
- Layer 25 alphabet latents: 21 of 114 identified latents have density >0.1, accounting for 20% of dense latents at that layer.
- Nullspace-aligned latents: 75% are high-density and account for 40% of high-density latents in the layer 25 SAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| zero-ablate the dense-latent subspace before retraining the SAE (ablation) | Gemma 2 2B layer 25 SAE · Gemma 2 2B layer 25 | number of high-density latents | Hurt |
| zero-ablate an equally sized non-dense-latent subspace before retraining the SAE (ablation) | Gemma 2 2B layer 25 SAE · Gemma 2 2B layer 25 | high-density latent distribution | No clear effect |
| allow negative latent activations and select TopK by absolute activation (AbsoluteTopK) (ablation) | TopK SAE · SAE antipodality | antipodality of dense latents | Hurt |
| freeze the RMSNorm scaling coefficient during nullspace-latent ablation (ablation) | Gemma 2 2B nullspace latents · Gemma 2 2B output | entropy change after ablation | Hurt |
| vary the SAE sparsity target L₀ or dictionary size (ablation) | Gemma 2 2B SAE · PC-aligned dense latents | number of PC-aligned latents | No clear effect |
Surprising findings
- AbsoluteTopK SAEs, which allow negative activations, eliminate antipodal dense latent pairs.
- Context-binding latents are specific within a context but explanations become vague when drawn across contexts.
Limitations
- Only about half of dense latents are explained; many remain uninterpreted.
- Interpretability analysis focuses on JumpReLU SAEs on Gemma 2 2B with a single dictionary size and sparsity.
- Dense latents may be noisy aggregations of sparse features or a non-aligned basis of true dense features.
Experimental setup
Datasets: OpenWebText C4 RedPajama Brown Corpus Neuronpedia activation densities
Metrics: activation density antipodality score Spearman correlation AUC-ROC fraction of correct flips entropy change principal angles
Baselines: AbsoluteTopK SAE sparse-latent ablation control
Builds on: sparse autoencoders TopK SAE JumpReLU SAE Gemma Scope
Closest papers by meaning
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.89)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.89)
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.89)
- Sparse Autoencoders, Again? (ICML 2025 · similarity 0.89)
- Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders (NeurIPS 2024 · similarity 0.88)
- Automatically Interpreting Millions of Features in Large Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.