Research map / Architectures and efficiency / RNNs and normalization
From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MP-SAE, a sparse autoencoder whose encoder unrolls matching pursuit into residual-guided greedy steps, to capture hierarchical, nonlinearly accessible, and multimodal features that standard SAEs miss.
Problem
Sparse autoencoders are motivated by the linear representation hypothesis (approximately orthogonal, linearly accessible concepts), but recent evidence shows neural representations contain hierarchical, nonlinear, and multi-dimensional structure; it is unclear whether SAEs can capture such features.
Approach
Re-contextualize matching pursuit as the SAE encoder: greedily select the feature best aligned with the current residual, subtract its contribution, and repeat for T steps, yielding conditionally orthogonal selections and nonlinearly accessible features, with adaptive sparsity at inference.
Key results
- MP-SAE achieves higher R^2 than standard SAEs at comparable sparsity across SigLIP, DINOv2, CLIP, and ViT representations (expansion factor 25).
- On the synthetic hierarchical tree, MP-SAE recovers both intra- and inter-level structure with no feature absorption, while Vanilla and BatchTopK suffer absorption and Matryoshka introduces negative sibling interference.
- MP-SAE's modality score distribution is flatter with mass in the midrange on CLIP joint image-text embeddings, whereas standard SAEs yield bimodal split dictionaries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use residual-guided sequential Matching Pursuit inference instead of a conventional SAE encoder | SAEs on synthetic hierarchical features · Synthetic hierarchical tree | hierarchical and intra-level structure recovery | Helped |
| Use residual-guided sequential Matching Pursuit inference instead of a conventional SAE encoder | SAEs on pretrained vision representations · ImageNet-1K representations from SigLIP, DINOv2, CLIP, and ViT | R² | Helped |
| Increase inference-time sparsity without retraining | MP-SAE · DINOv2 representations | reconstruction error | Helped |
| Use residual-guided sequential inference rather than a linear SAE encoder | SAEs on joint image-text embeddings · COCO image and caption embeddings | modality-shared feature recovery | Helped |
| Use MP-SAE rather than Matryoshka SAE | SAEs on hierarchical synthetic features · Synthetic tree, child activation mean 0.5 and variance 1/400 | absorption score: 4.8e-02 vs 7.3e-02 | Hurt |
| Use MP-SAE rather than Vanilla SAE | SAEs on hierarchical synthetic features · Synthetic tree, child activation mean 0.5 and variance 1/400 | absorption score: 7.0e-01 vs 7.3e-02 | Helped |
| Use MP-SAE rather than BatchTopK SAE | SAEs on hierarchical synthetic features · Synthetic tree, child activation mean 0.5 and variance 1/400 | absorption score: 5.0e-01 vs 7.3e-02 | Helped |
| Use MP-SAE rather than Matryoshka SAE | SAEs on hierarchical synthetic features · Synthetic tree, child activation mean 1.0 and variance 1/400 | absorption score: 2.1e-01 vs 1.3e-03 | Helped |
| Use MP-SAE rather than Vanilla SAE | SAEs on hierarchical synthetic features · Synthetic tree, child activation mean 1.0 and variance 1/400 | absorption score: 4.5e-01 vs 1.3e-03 | Helped |
| Use MP-SAE rather than BatchTopK SAE | SAEs on hierarchical synthetic features · Synthetic tree, child activation mean 1.0 and variance 1/400 | absorption score: 4.8e-01 vs 1.3e-03 | Helped |
| Increase within-level feature correlation | MP-SAE on the synthetic hierarchy · Synthetic hierarchical tree | flat and hierarchical structure recovery | Mixed |
| Vary child activation mean and variance | MP-SAE on the synthetic hierarchy · Synthetic hierarchical tree | absorption score | Mixed |
Surprising findings
- MP-SAE learns dictionaries with higher global Babel scores (more interference) yet selects less-interfering feature subsets at inference.
- Standard TopK SAEs degrade in reconstruction when inference-time sparsity differs from training sparsity.
Limitations
- Matching pursuit is greedy and lacks global optimality, may be brittle under extreme noise.
- Inductive bias may be poorly matched to flat or entangled regimes or categorical structure.
- On the original Matryoshka benchmark with perfectly correlated firings, MP-SAE behavior is unstable.
Experimental setup
Datasets: ImageNet-1K COCO synthetic hierarchical tree data
Benchmarks: COCO
Metrics: R^2 reconstruction Babel function effective rank modality score flat MSE hierarchical MSE absorption score
Baselines: Vanilla SAE (ReLU) BatchTopK SAE Matryoshka SAE TopK SAE
Builds on: Matching Pursuit Matryoshka SAE BatchTopK SAE Linear Representation Hypothesis
Closest papers by meaning
- Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry (NeurIPS 2025 · similarity 0.92)
- Mechanistic Interpretability with Sparse Autoencoder Neural Operators (NeurIPS 2026 · similarity 0.91)
- Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders (ICML 2025 · similarity 0.90)
- Interpreting CLIP with Hierarchical Sparse Autoencoders (ICML 2025 · similarity 0.90)
- Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression (NeurIPS 2026 · similarity 0.89)
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.