Research map / Vision-language and multimodal / CLIP and zero-shot
VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VL-SAE, a sparse autoencoder with a distance-based top-k encoder and modality-specific decoders that maps vision and language representations of CLIP-style and large VLMs into a unified concept set, used to interpret alignment and improve zero-shot classification and POPE hallucination reduction.
Problem
Vision-language alignment in VLMs is hard to interpret because SAEs trained separately on vision and language representations yield mismatched concept sets, so semantically similar image and text representations do not co-activate the same neurons.
Approach
Use an auxiliary autoencoder to make LVLM alignment explicit via InfoNCE, then train a shared SAE with a distance-based (Euclidean on normalized vectors) top-k encoder and separate decoders per modality so semantically similar image-text representations co-activate the same neurons.
Key results
- On OpenCLIP ViT-H/14, VL-SAE-based concept-level alignment raises mean zero-shot accuracy across 14 datasets from 76.9 to 77.8.
- On POPE with LLaVA-1.5 (random setting), VL-SAE reaches 87.07 accuracy versus 85.53 for VCD and 82.93 for regular decoding.
- Intra-Similarity of concepts rises from 0.1890 (standard SAE) to 0.2134 on OpenCLIP-ViT-H/14 and Inter-Similarity falls from 0.1688 to 0.1149.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add concept-level alignment with VL-SAE | OpenCLIP · Zero-shot image classification (Mean Acc.) | accuracy: 68.7 -> 69.5; 69.8 -> 70.4; 72.2 -> 72.9; 76.9 -> 77.8 | Helped |
| Use VL-SAE decoding instead of VCD | LLaVA1.5 · POPE Random, Popular, and Adversarial | accuracy: 85.53 -> 87.07; 83.63 -> 85.87; 81.10 -> 83.60 | Helped |
| Use VL-SAE decoding instead of VCD | Qwen-VL · POPE Random, Popular, and Adversarial | accuracy: 86.33 -> 86.47; 85.73 -> 85.97; 83.77 -> 86.10 | Mixed |
| Use VL-SAE decoding instead of VCD | LLaVA1.5 and Qwen-VL · CHAIR subset | CHAIR_S: 55.0 -> 47.8; 42.6 -> 39.6 | Mixed |
| Replace the standard encoder with a cosine-based encoder (ablation) | VL-SAE on OpenCLIP-ViT-H/14 · Concept quality | Intra-Similarity: 0.1890 -> 0.1891 | Helped |
| Replace the cosine-based encoder with a distance-based encoder (ablation) | VL-SAE on OpenCLIP-ViT-H/14 · Concept quality | Intra-Similarity: 0.1891 -> 0.2016 | Helped |
| Add modality-specific decoders (ablation) | VL-SAE on OpenCLIP-ViT-H/14 · Concept quality | Intra-Similarity: 0.2016 -> 0.2134 | Helped |
| Remove the auxiliary autoencoder (ablation) | VL-SAE on LLaVA1.5 · Concept quality | Intra-Similarity: 0.2092 -> 0.2084 | Hurt |
| Increase CC3M training data from 10% to 100% (ablation) | VL-SAE · Concept quality | Intra-Similarity: 0.2029 -> 0.2299 | Helped |
| Use Top-K instead of L1 sparsification (ablation) | VL-SAE · Concept quality | Intra-Similarity: 0.2142 -> 0.2442 | Helped |
| Use task-agnostic alpha_c=0.1 instead of alpha_c=0 (ablation) | OpenCLIP ViT-L and ViT-H · Zero-shot image classification | accuracy: 72.2 -> 72.4; 76.9 -> 77.2 | Helped |
| Use task-specific alpha_c instead of alpha_c=0 (ablation) | OpenCLIP ViT-L and ViT-H · Zero-shot image classification | accuracy: 72.2 -> 72.9; 76.9 -> 77.8 | Helped |
Surprising findings
- Fine-tuning CLIP ViT-B/16 on CC3M does not improve downstream accuracy (69.7 vs 69.8 base), suggesting gains come from the concept mapping rather than the data.
Limitations
- Dead neurons and high-frequency neurons affect interpretation results.
- Concept enhancement strategy is simple and does not fully exploit the unified concept set.
- CLIP-based similarity is an imperfect metric for concept quality.
Experimental setup
Datasets: CC3M CHAIR POPE
Benchmarks: POPE CHAIR zero-shot image classification datasets (Caltech101, CIFAR10/100, DTD, EuroSAT, etc.)
Metrics: Intra-Similarity Inter-Similarity accuracy F1 score CHAIR_S CHAIR_I Recall
Baselines: SAE-D SAE-S VCD Regular decoding L1 sparsification
Builds on: sparse autoencoder top-k sparse autoencoder contrastive decoding InfoNCE
Closest papers by meaning
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.91)
- NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.91)
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.90)
- Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge (CVPR 2025 · similarity 0.89)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- LVLM-Aided Alignment of Task-Specific Vision Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.