Research map / Architectures and efficiency / RNNs and normalization
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
David Chanin, Adrià Garriga-Alonso
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| set SAE L0 to the true L0 instead of too low (1.8 vs 2) (ablation) | BatchTopK SAE · Toy model (5 features) | feature recovery: L0 1.8 -> L0 2 | Improved |
| use ground-truth SAE instead of trained low-L0 SAE | SAE at L0=5 · Toy model reconstruction | MSE: 2.73 -> 4.88 | Hurt |
| start training from too-high L0 (20) then decay to correct L0 (11) (ablation) | BatchTopK SAE · Toy model (50 features) | feature recovery: L0 20 -> 11 | No clear effect |
| start training from too-low L0 (2) then raise to correct L0 (11) (ablation) | BatchTopK SAE · Toy model (50 features) | feature recovery: L0 2 -> 11 | Hurt |
| train SAE with L0 too high (18) instead of correct (11) (ablation) | BatchTopK SAE · Toy model (50 features) | feature recovery: L0 11 -> 18 | Hurt |
| train SAE with L0 too low (5) instead of correct (11) (ablation) | BatchTopK SAE · Toy model (50 features) | feature recovery: L0 11 -> 5 | Hurt |
| use JumpReLU instead of BatchTopK at high L0 | SAE · Gemma-2-2b layer 12 | k-sparse probing F1 | Improved |
| use L0 near 200 (c_dec elbow) instead of lower L0 | BatchTopK SAE · Gemma-2-2b layer 5 and Llama-3.2-1b layer 7 | k-sparse probing F1: elbow around L0 200 | Improved |
Closest papers by meaning
- Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? (NeurIPS 2026 · similarity 0.91)
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.90)
- Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders (NeurIPS 2024 · similarity 0.90)
- Sparse Autoencoders Do Not Find Canonical Units of Analysis (ICLR 2025 · similarity 0.89)
- Low-Rank Adapting Models for Sparse Autoencoders (ICML 2025 · similarity 0.88)
- Sparse Autoencoders, Again? (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
