Research map / Architectures and efficiency / RNNs and normalization

SASA: Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Sparse dictionary representations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Subspace-Aware Sparse Autoencoders (SASA), which replace single-vector SAE decoders with block decoder subspaces, Top-s group gating, and a nuclear-norm rank regularizer, to reduce feature splitting and sample complexity.

Problem

Standard sparse autoencoders assign one decoder direction per latent, which implicitly assumes 1-D features; this mismatch with multi-dimensional feature subspaces causes feature splitting and high sample complexity for training on LLM activations.

Approach

Replace single-vector decoders with learned decoder subspaces (blocks), enforce sparsity at the block level via Top-s group gating, and regularize each block's reconstruction map with a nuclear norm so a single group can capture an entire multi-dimensional feature.

Key results

• On Mistral-7B, SASA with 500M tokens achieves 98.6% fraction explained variance vs 96.7% for standard SAE trained on 1B tokens.
• On GPT-2 layer 7, TopK-SASA reduces mean absorption to 0.046 vs 0.114 for TopK SAE, with AutoInterp 0.833 vs 0.831.
• On GPT-2, SASA full fractional absorption drops to 4.7% vs 49.6% for the standard SAE with half the token budget.

Surprising findings

• Standard SAE on Mistral-7B uses l0 of 285 while SASA uses 80 at similar reconstruction quality.

Limitations

• Remaining compute is still too high for very large LLMs.
• Learning a feature subspace does not yield an interpretable coordinate system within it; relies on manual inspection.
• Theory assumes idealized population objective on a single-feature slice and separable block cost.

Experimental setup

Datasets: web text corpus (residual-stream activations) synthetic manifolds (circle, sphere, helix) first-letter benchmark

Benchmarks: KL score CE score fraction of explained variance first-letter feature absorption benchmark AutoInterp

Metrics: KL score CE score fraction explained variance l0 sparsity mean absorption fraction absorption AutoInterp score

Baselines: ReLU SAE Gated SAE JumpReLU SAE TopK SAE BatchTopK SAE Mistral SAE

Builds on: sparse autoencoders TopK SAE BatchTopK SAE JumpReLU SAE Gated SAE

Closest papers by meaning

• A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders (NeurIPS 2025 · similarity 0.90)
• Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.90)
• Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders (NeurIPS 2024 · similarity 0.90)
• Sparse Autoencoders, Again? (ICML 2025 · similarity 0.90)
• SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability (ICML 2025 · similarity 0.89)
• Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.89)