Research map / Architectures and efficiency / RNNs and normalization

Mechanistic Interpretability with Sparse Autoencoder Neural Operators

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Sparse dictionary representations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces sparse autoencoder neural operators (SAE-NOs), instantiated as SAE-FNOs, which parameterize SAE concepts as functions with joint concept and domain sparsity, showing more stable, localized, efficient concepts and resolution generalization on vision data.

Problem

Standard sparse autoencoders represent concepts as vectors with scalar activations, capturing concept presence but not how or where concepts are expressed across a spatial domain, and they fail to generalize across discretizations and resolutions.

Approach

Parameterize SAE concepts as functions via Fourier neural operators, with concept sparsity (group sparsity selecting active concepts) and domain sparsity (element-wise sparsity within each concept's feature map), plus a lifting operator that acts as a preconditioner.

Key results

• SAE-FNO keeps concept structure stable across sparsity levels, whereas SAE-MLP concepts change drastically as k increases (Hoyer-based and identity-consistency analysis).
• SAE-FNO trained at 28x28 continues to represent data accurately at higher resolutions, while standard SAE reconstruction error increases sharply beyond training discretization.
• Lifting acts as a preconditioner, accelerating dictionary recovery in synthetic sparse generative model experiments.

Surprising findings

• Even without concept sparsity (k=p), domain sparsity alone gives SAE-FNO concepts with similar structural properties, while SAE-MLP degrades into noise-like patterns.

Limitations

• Higher computational and memory cost than SAE-MLP due to Fourier operations and feature-map representations.
• Two extra hyperparameters (number of Fourier modes and domain sparsity) to tune.
• Empirical study limited to vision datasets; not tested on LLM or neural operator foundation model activations.

Experimental setup

Datasets: MNIST CIFAR-10 ImageNet synthetic sparse generative data synthetic 1D time-series moving MNIST

Benchmarks: MNIST CIFAR-10 ImageNet

Metrics: Hoyer score identity consistency R^2 L0 concept utilization PSNR dictionary recovery error

Baselines: SAE-MLP SAE-CNN TopK SAE BatchTopK SAE Matryoshka SAE

Builds on: TopK SAE BatchTopK SAE Matryoshka SAE Fourier neural operator linear representation hypothesis

Closest papers by meaning

• Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry (NeurIPS 2025 · similarity 0.92)
• From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit (NeurIPS 2025 · similarity 0.91)
• Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment (ICML 2025 · similarity 0.90)
• Learning Multi-Level Features with Matryoshka Sparse Autoencoders (ICML 2025 · similarity 0.90)
• Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders (ICML 2025 · similarity 0.89)
• Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? (NeurIPS 2026 · similarity 0.89)