Research map / Architectures and efficiency / RNNs and normalization

Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Sparse dictionary representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a hierarchical sparse autoencoder (H-SAE) where each top-level SAE feature gates a low-dimensional expert SAE, improving reconstruction and reducing feature absorption and cross-lingual redundancy in Gemma 2-2B residual streams.

Problem

Standard sparse autoencoders learn flat dictionaries that ignore semantic relationships between concepts, leading to poor reconstruction-interpretability tradeoffs, feature splitting, and feature absorption as dictionary size grows.

Approach

Each top-level SAE atom owns a low-dimensional projected subspace with its own low-level SAE that is only activated when the parent atom is active, implementing a parent-child (sum of high-level and contextual low-level vectors) structure as a mixture-of-experts with a bi-orthogonality penalty.

Key results

• H-SAE with 64 sublatents per expert matches a standard SAE with 32 top-level features while using about 1/4 the compute cost.
• H-SAE improves first-letter classification performance and slows the growth of absorption as width increases relative to the baseline SAE.
• H-SAE yields more similar top-8 feature sets for the same token across English, French, Spanish, and German than the baseline SAE.

Surprising findings

• Ablations on the unembedding matrix suggest the orthogonality and l1 regularizers are not necessary for interpretability or hierarchy, despite being kept in final runs.
• Without whitening (causal inner product) the unembedding SAE learns no meaningful features.

Limitations

• Reconstruction remains imperfect and some low-level features are hard to interpret.
• Interpretability evaluation relies on targeted proxies (absorption, redundancy) rather than comprehensive automated interpretability metrics.
• Orthogonality and l1 terms lack an empirical measure of compositionality and their necessity is unclear.

Experimental setup

Datasets: Wikipedia 20231101 dump SAEBench first-letter classification

Benchmarks: SAEBench CE loss score SAEBench first-letter classification cross-lingual feature redundancy test

Metrics: 1 - explained variance normalized CE loss score absorption fraction symmetric set difference of top-8 features across languages

Baselines: TopK SAE Gemma Scope JumpReLU SAE Switch SAE Matryoshka SAE

Builds on: TopK SAE Matryoshka SAE Switch SAE Gemma Scope

Closest papers by meaning

• Learning Multi-Level Features with Matryoshka Sparse Autoencoders (ICML 2025 · similarity 0.90)
• Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders (ICML 2025 · similarity 0.89)
• A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders (NeurIPS 2025 · similarity 0.89)
• Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.89)
• Cascaded Sparse Autoencoders LearnMulti-Level Visual Concepts in Multimodal LLMs (NeurIPS 2026 · similarity 0.89)
• Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.89)