Research map / Architectures and efficiency / RNNs and normalization
The Price of Amortized inference in Sparse Autoencoders
Wenjie Sun, Di Wang, Lijie Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase sparsity penalty in standard and TopK SAEs | SAE training · Gemma-2-2B layer 12 (SAEBench) | NMSE and dead rate | Hurt |
| continue training to shrink amortization gap | Standard and TopK SAEs · Gemma-2-2B layer 12 | monosemanticity metrics | No clear effect |
| add semi-amortized ISTA refinement (ablation) | TopK SAE · Pythia-160M layer 8 | NMSE: 1.499 -> 0.087 | Improved |
| use non-amortized ISTA inference (ablation) | GatedSAE · Pythia-160M layer 8 | NMSE: 1.947 -> 0.000 | Improved |
| add semi-amortized refinement (ablation) | BatchTopK SAE · Pythia-160M layer 8 | dead latent rate: 0.316 -> 0.000 | Improved |
| use non-amortized ISTA inference (ablation) | Standard SAE · Pythia-160M layer 8 | absorption rate: 0.144 -> 0.283 | Hurt |
| use non-amortized ISTA inference (ablation) | TopK SAE · Pythia-160M layer 8 | absorption rate: 0.225 -> 0.452 | Hurt |
| add semi-amortized refinement (ablation) | TopK SAE · TPP Pythia-160M layer 8 | selectivity: 0.245 -> 0.508 | Improved |
| add semi-amortized refinement (ablation) | TopK SAE · GIS Pythia-160M layer 8 | intervention score (KL 1.00): 1.412 -> 1.968 | Improved |
| increase ISTA refinement steps from 5 to 50 (ablation) | semi-amortized BatchTopK SAE · Pythia-160M layer 8 | NMSE: 0.477 -> 0.046 | Improved |
| increase ISTA refinement steps (ablation) | semi-amortized BatchTopK SAE · Pythia-160M layer 8 | dead rate and feature splitting: dead rate 0.000 throughout | No clear effect |
| group latents by angular variance with group encoders (LocA-SAE) | amortized SAE · Pythia-160M layer 8 | dead rate: 0.000 vs 0.316 (BatchTopK full-amortized) | Improved |
Closest papers by meaning
- Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders (ICML 2025 · similarity 0.90)
- SASA: Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability (NeurIPS 2026 · similarity 0.89)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.88)
- Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words (ICLR 2025 · similarity 0.88)
- Sparse Autoencoders Do Not Find Canonical Units of Analysis (ICLR 2025 · similarity 0.88)
- Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
