Research map / Probabilistic and scientific modeling / Variational and generative sampling
Sparse Autoencoders, Again?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VAEase, a VAE variant that gates the latent mean by (1 - posterior variance) so inactive dimensions are zeroed, giving input-adaptive sparse autoencoding without extra hyperparameters, with theory on manifold dimension recovery and local minima.
Problem
Sparse autoencoders need sensitive hyperparameters and nonconvex penalties, while VAEs are hyperparameter-free but yield fixed (input-independent) sparsity patterns, limiting their use for manifold dimension estimation and LLM activation sparse coding.
Approach
Replace the VAE decoder input with (1 - sigma_z) ⊙ z, using the posterior variance network only as a gate that suppresses noisy inactive latent dimensions, so the decoder can keep all weights nonzero and sparsity becomes input-adaptive, with no added parameters versus a vanilla VAE.
Key results
- On the synthetic nonlinear dataset with ground-truth dims {5,5,10,10}, VAEase estimates {5,5,11,11} while SAE-l1, SAE-log, SAE-T_k and VAE are far off (e.g., VAE 13-21 per manifold).
- On LLM activations, VAEase achieves RE 39.5 and average active dims 22.5, beating SAE-l1 (46.2, 38.2) and SAE-T_k (45.1, 30.0).
- On pseudo-MNIST with true upper bound 16, VAEase estimates 14.93 active dims versus 105.94 (DM-NB) and 169.81 (DM-FLIPD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VAEase instead of a standard VAE | sparse autoencoding · Linear Subspaces | active-dimension estimation relative to ground truth: GT 4, 4, 4; VAE 13, 13, 13; VAEase 5, 4, 4 | Helped |
| Use VAEase instead of a standard VAE | sparse autoencoding · Nonlinear Manifolds | active-dimension estimation relative to ground truth: GT 5, 5, 10, 10; VAE 17, 21, 20, 19; VAEase 5, 5, 11, 11 | Helped |
| Use VAEase instead of VAE | sparse autoencoding · MNIST | reconstruction error: 9.81 -> 9.71 | Helped |
| Use VAEase instead of SAE-T_k | sparse autoencoding · MNIST | average active dimensions: 16.0 -> 16.2 | Mixed |
| Use VAEase instead of SAE-T_k | sparse autoencoding · FashionMNIST | reconstruction error: 8.11 -> 8.39 | Hurt |
| Use VAEase instead of SAE-T_k | sparse autoencoding · FashionMNIST | average active dimensions: 15.0 -> 12.4 | Helped |
| Use VAEase instead of SAE-T_k | sparse autoencoding · LLM-Act | reconstruction error: 45.1 -> 39.5 | Helped |
| Use VAEase instead of SAE-T_k | sparse autoencoding · LLM-Act | average active dimensions: 30.0 -> 22.5 | Helped |
| Use VAEase instead of SAE-T_k | sparse autoencoding · Text-Emb | reconstruction error: 0.217 -> 0.187 | Helped |
| Use VAEase instead of SAE-T_k | sparse autoencoding · Text-Emb | average active dimensions: 30.0 -> 16.7 | Helped |
| Estimate dimension with VAEase rather than diffusion post-processing | manifold dimension estimation · Pseudo-MNIST | estimated manifold dimension: GT 16; DM (NB) 105.94; DM (FLIPD) 169.81; VAEase 14.93 | Helped |
| Add variance-based gating of the latent code to the VAE decoder | VAEase · union of low-dimensional manifolds | adaptive sparsity aligned to manifold dimensions | Helped |
Surprising findings
- Concatenating the posterior variance with the latent code as decoder input is shown to fail, as the decoder can ignore z and reconstruct from sigma_z, driving the loss toward minus infinity.
- SAE-T_k did not yield lower reconstruction error even with higher active-dimension counts on LLM and text data.
Limitations
- Theoretical guarantees are proven for linear or simplified decoder settings and Lipschitz assumptions; the general nonlinear case is supported mainly empirically.
- Ground-truth manifold dimensions are unavailable for real data, so evaluation on MNIST and LLM activations relies on reconstruction-vs-active-dimension tradeoffs and proxies.
Experimental setup
Datasets: synthetic union of linear subspaces synthetic union of nonlinear manifolds MNIST FashionMNIST Pile-10k Yelp reviews (HypotheSAEs embeddings) pseudo-MNIST (GAN-generated)
Benchmarks: synthetic manifold dimension estimation MNIST FashionMNIST LLM activation sparse coding (Pythia/Pile-10k) Yelp text embeddings pseudo-MNIST manifold dimension
Metrics: average active dimensions reconstruction error estimated manifold dimension
Baselines: SAE-l1 SAE-log SAE-T_k (top-k SAE) VAE Normalizing-free diffusion NB estimator FLIPD
Builds on: variational autoencoder sparse autoencoder conditional VAE Lipschitz VAE sparsity analysis
Closest papers by meaning
- Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders (NeurIPS 2024 · similarity 0.91)
- Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? (NeurIPS 2026 · similarity 0.91)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.91)
- Low-Rank Adapting Models for Sparse Autoencoders (ICML 2025 · similarity 0.90)
- Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.