Research map / Vision-language and multimodal / CLIP and zero-shot

Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering

NeurIPS 2026 · Vision-language and multimodal › CLIP and zero-shot · approach: Open-vocabulary alignment from VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Visual Sparse Steering (VS2), a label-free method that steers frozen CLIP embeddings by amplifying active top-k SAE features, with an FVU-gated fallback and a VS2++ oracle study of selective amplification.

Problem

Whether sparse autoencoder features learned from unlabeled activations can serve as an actionable, label-free intervention space to improve zero-shot classification of frozen vision-language models, and when such steering is reliable.

Approach

Train a top-k SAE on unlabeled CLIP CLS activations; at test time amplify each input's active sparse features by a factor, decode the difference from the un-amplified reconstruction, add it to the embedding and renormalize. This equals centroid-deviation steering with a residual bounded by FVU, which gates steering per sample.

Key results

• VS2 improves zero-shot CLIP ViT-B/16 on CIFAR-100 by +4.12% (68.08 vs 63.96) with under 0.1% extra inference compute.
• Oracle-label VS2++ reaches +21.44% over zero-shot on CIFAR-100 ViT-B/16 (85.40), indicating large headroom from selective feature amplification.
• With a generalized SAE, FVU gating at q=0.90 keeps CUB-200 at zero-shot accuracy by disabling steering (0% coverage), while ungated steering drops by up to 5.84%.

Surprising findings

• Plain SAE reconstruction without amplification usually hurts zero-shot accuracy, e.g. -8.12% on Tiny-ImageNet ViT-B/16 for all-layer reconstruction.
• Zeroing or negating top-k features collapses accuracy to near chance (e.g. 0.06% on CIFAR-100 for negation), though this is in the amplified-reconstruction variant rather than VS2 itself.

Limitations

• Requires an offline SAE trained on target-distribution activations; poor generalization under shift degrades steering.
• Evaluated only on CLIP backbones and nine classification datasets; ImageNet-scale and non-CLIP VLMs not tested.
• The VS2++ oracle uses ground-truth labels and is an analysis tool, not a deployable method; non-oracle gains are smaller and noisy.

Experimental setup

Datasets: CIFAR-100 CUB-200 Tiny-ImageNet Aircraft Food101 Flower102 Caltech101 SUN EuroSAT

Benchmarks: CIFAR-100 CUB-200 Tiny-ImageNet Aircraft Food101 Flower102 Caltech101 SUN EuroSAT

Metrics: top-1 accuracy top-5 accuracy FVU coverage GFLOPs

Baselines: CLIP zero-shot SAE reconstruction SAE amplified reconstruction TPT C-TPT Diff-TPT SpLiCE weighted RAG CLIP steering

Builds on: top-k sparse autoencoder CLIP test-time prompt tuning SpLiCE

Closest papers by meaning

• Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control (ICLR 2025 · similarity 0.88)
• Sparse autoencoders reveal selective remapping of visual concepts during adaptation (ICLR 2025 · similarity 0.88)
• Cascaded Sparse Autoencoders LearnMulti-Level Visual Concepts in Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
• From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit (NeurIPS 2025 · similarity 0.87)
• Scaling and evaluating sparse autoencoders (ICLR 2025 · similarity 0.87)