Research map / Large language models / Language, safety and interpretability
Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates whether sparse autoencoder (SAE) probes beat strong baselines for LLM activation probing across 113 datasets and hard regimes (data scarcity, class imbalance, label noise, covariate shift), finding they generally do not.
Problem
Whether SAEs provide a practical, downstream benefit for interpreting LLM activations, given the lack of ground truth for concepts and reliance on proxy metrics; specifically whether SAE-based linear probes beat standard probing baselines in difficult regimes.
Approach
Introduces a "quiver of arrows" evaluation that adds SAE probes as one option among baseline probing methods and selects among them by validation AUC, emulating a practitioner's toolkit; SAE probes select top-k latents by class-mean difference and train L1 logistic regression on them.
Key results
- SAE probes underperform logistic regression baseline on average across datasets in each regime; adding SAE probes to the quiver gives no average improvement in data scarcity, class imbalance, or label noise.
- SAE probes outperform baselines on only 2.2% of datasets with last-token probing; max-pooled SAE probes win 19.6% vs last-token baseline but drop to 8.7% when attention-pooled baselines are included.
- On CoLA, the SAE latent 369585 classifier outperforms the dense SAE probe and logistic regression on mislabeled examples, but baseline logistic regression can identify the same dataset errors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SAE probes to the baseline-method quiver | Gemma-2-9B probes under standard conditions · 113 probing datasets | test AUC | Hurt |
| Add SAE probes to the baseline-method quiver | Gemma-2-9B probes in data scarcity, class imbalance, and label noise · 113 probing datasets | test AUC | No clear effect |
| Use SAE probes instead of baseline probes under covariate shift | Gemma-2-9B probes · Covariate-shifted datasets | AUC | Hurt |
| Prune the SAE probe from k=8 to k=1 using o1's latent relevance ranking (ablation) | SAE probes · 66_living-room OOD | AUC: 0.024 increase in AUC | Helped |
| Prune the SAE probe from k=8 to k=1 using o1's latent relevance ranking (ablation) | SAE probes · 90_glue_qnli OOD | AUC: 0.052 increase in AUC | Helped |
| Prune the SAE probe from k=8 to k=1 using o1's latent relevance ranking (ablation) | SAE probes · 7_hist_fig_ispolitician OOD | AUC: 0.077 | Hurt |
| Replace last-token baseline probing with max-pooled SAE probing | Multi-token probes · 60 random probing datasets | win rate: 2.2% -> 19.6% | Helped |
| Add attention-pooled baselines and select pooled or last-token strategies with the quiver | Multi-token SAE probes · 60 random probing datasets | SAE probe win rate: 19.6% -> 8.7% | Hurt |
| Use more recent SAE architectures | Gemma-2-2B layer 12 SAE probes · Standard conditions | test AUC | No clear effect |
| Binarize SAE latents (ablation) | SAE probes · Probing datasets | test AUC | No clear effect |
| Use Llama-3.1-8B SAE probes instead of baseline probes | Llama-3.1-8B probes · Standard conditions, data scarcity, class imbalance, and label noise | test AUC | No clear effect |
Surprising findings
- Initial SAE-based findings on spurious correlations (AI vs human text, CoLA label errors) could be reproduced with simple logistic regression baselines.
- Max-pooled SAE probes that appear to beat baselines lose most of their advantage once attention-pooled baselines are included.
Limitations
- Probing performance is a proxy for SAE utility, and even a ground-truth feature basis might offer only a mild inductive bias advantage.
- Further optimization of SAE probe baselines (multi-token, binarization, larger k) might still beat baselines; binarization did not significantly help in the authors' tests.
- Quiver of arrows uses test AUC for choosing k and SAE in some analyses, a slight test-set leakage acknowledged by the authors.
Experimental setup
Datasets: 113 binary classification datasets (e.g., GLUE tasks, WikiData occupation, headlines, AI-generated text detection, CoLA) GLUE-X datasets The Pile
Benchmarks: 113 binary classification probing datasets GLUE-X covariate shift datasets
Metrics: AUC quiver of arrows test AUC win rate
Baselines: logistic regression PCA regression k-nearest neighbors XGBoost multilayer perceptron attention-pooled probes max-pooled SAE probes
Builds on: JumpReLU TopK Gemma Scope Llama Scope XGBoost
Closest papers by meaning
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.93)
- Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders (NeurIPS 2024 · similarity 0.93)
- Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? (NeurIPS 2026 · similarity 0.92)
- Automatically Interpreting Millions of Features in Large Language Models (ICML 2025 · similarity 0.92)
- Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words (ICLR 2025 · similarity 0.92)
- Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.