Research map / Architectures and efficiency / RNNs and normalization
Taming Polysemanticity in LLMs: Theory-Grounded Feature Recovery via Sparse Autoencoders
Siyu Chen, Heejune Sheen, Xuyuan Xiong, Tianhao Wang, Zhuoran Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TopK with K mismatched to data sparsity (K=20 to K=50) (ablation) | TopK SAE · Imbalanced synthetic (n=128, d=42, M=8192) | FRR (tau_align>=0.9): 98.4% -> 23.4% | Hurt |
| use GBA with full frequency coverage instead of TopK | SAE training · Imbalanced synthetic (n=128, d=42, M=8192) | FRR (tau_align>=0.9): 23.4% -> 100.0% | Improved |
| misspecify GBA frequency coverage (ablation) | GBA · Imbalanced synthetic (n=128, d=42, M=8192) | FRR (tau_align>=0.8): 100.0% -> 38.3% | Hurt |
| use GBA instead of TopK for cross-seed consistency | SAE training · Qwen2.5-1.5B Github L26, top-10% neurons | Fraction MCS>0.9: 0.0317 -> 0.0366 | Improved |
| add neuron grouping (GBA vs single-group BA) (ablation) | Bias adaptation · Qwen2.5-1.5B Pile reconstruction vs sparsity | Reconstruction loss at equal sparsity: significantly outperforming both the L1 penalty method and the non-grouped variant BA | Improved |
| compare GBA with TopK on reconstruction-sparsity frontier | GBA · Qwen2.5-1.5B Pile reconstruction vs sparsity | Reconstruction loss: comparable to TopK's best curve | No clear effect |
| use GBA instead of TopK on SAEBench explained variance | SAE training · SAEBench (Gemma2-2B, layer 12, L0 300-400) | Explained variance: 0.859 -> 0.902 | Improved |
| use GBA instead of TopK on absorption score | SAE training · SAEBench (Gemma2-2B, layer 12, L0 300-400) | Absorption score: 0.0269 -> 0.0041 | Improved |
| use GBA instead of TopK on alive fraction | SAE training · SAEBench (Gemma2-2B, layer 12, L0 300-400) | Alive fraction: 0.872 -> 0.970 | Improved |
| use GBA instead of TopK on TPP and SCR | SAE training · SAEBench (Gemma2-2B, layer 12, L0 300-400) | TPP / SCR: 0.213 -> 0.184 (TPP); 0.332 -> 0.296 (SCR) | Hurt |
| increase number of groups K to 10 or more (ablation) | GBA · Github Layer 26 | Reconstruction loss and sparsity: stabilize when K >= 10 | No clear effect |
| increase highest target frequency HTF toward 0.5 (ablation) | GBA · Github Layer 26 | Reconstruction loss: HTF 0.05 gives higher reconstruction loss; stabilizes at HTF = 0.5 | Improved |
Closest papers by meaning
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.89)
- Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders (NeurIPS 2024 · similarity 0.89)
- Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.89)
- Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations (ICML 2025 · similarity 0.89)
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
