Research map / Architectures and efficiency / RNNs and normalization

Cascaded Sparse Autoencoders LearnMulti-Level Visual Concepts in Multimodal LLMs

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Sparse dictionary representations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAE++, a cascaded sparse autoencoder that trains a second-level SAE on the first-level SAE's decoder atoms to learn hierarchical "concepts of concepts" in MLLM visual representations, outperforming flat and Matryoshka/stacked SAEs on hierarchical mono-semanticity and concept steering.

Problem

Existing SAEs recover flat feature dictionaries; Matryoshka SAEs create shared-prefix coupling and stacked SAEs create sparse bottlenecks, so explicit multi-level visual concept hierarchies in MLLMs are hard to learn and interpret.

Approach

A second-level SAE is trained directly on the columns of the first-level SAE's decoder (which live in the same activation space as MLLM activations), so higher-level concepts are sparse codes over concept directions rather than over sample codes or nested prefixes; the two levels are trained jointly with a weighting alpha.

Key results

• SAE++ achieves the best HMS mean in all 12 model-dataset settings; e.g., Qwen3 COCO HMS mean 0.980 vs 0.461 for the strongest baseline (MSAE).
• On COCO steering with Qwen3-VL-4B, SAE++ reaches 71.0% average insertion success and 40.1% suppression success across four judges, versus 40.9% and 23.6% for MSAE.
• Ablation on ImageNet: full SAE++ HMS median 0.9999 vs 0.8587 for HAC post-hoc clustering and 0.3367 for stacked SAE.

Surprising findings

• Stacked SAEs perform substantially worse than most flat baselines on HMS, consistent with the claimed sparse-bottleneck failure.

Limitations

• Quality depends on the Level-1 SAE; noisy, dead, or polysemantic atoms degrade the hierarchy.
• HMS measures visual coherence only and does not penalize duplicated concepts; evaluation is limited to two-level hierarchy on four datasets.
• Steering evaluated on a limited set of Level-2 concepts with LLM judges only, not causal or safety evaluation.

Experimental setup

Datasets: Color ImageNet iNaturalist COCO

Benchmarks: Color ImageNet iNaturalist COCO

Metrics: HMS mean HMS median concept steering insertion success rate concept steering suppression success rate

Baselines: BatchTopK SAE TopK SAE ReLU SAE P-Annealing SAE Gated SAE JumpReLU SAE Matryoshka SAE Stacked SAE HAC post-hoc clustering spectral clustering

Builds on: Matryoshka SAE TopK SAE JumpReLU SAE Gated SAE

Closest papers by meaning

• Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment (ICML 2025 · similarity 0.90)
• Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• Learning Multi-Level Features with Matryoshka Sparse Autoencoders (ICML 2025 · similarity 0.89)
• Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures (NeurIPS 2026 · similarity 0.89)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (ICML 2025 · similarity 0.89)