Research map / Vision-language and multimodal / Human and embodied modeling
Interpretability Transfer from Language to Vision via Sparse Autoencoders
Alexey Kravets, Da Li, Chuan Li, Da Chen, Vinay P. Namboodiri
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SAE reconstruction loss to projector training (ablation) | VISTA DINOv2, Gemma-2-2B-it · COCO steering, remove | mean score: 0.22 -> 1.43 | Improved |
| add SAE reconstruction loss to projector training (ablation) | VISTA DINOv2, Gemma-2-2B-it · COCO steering, replace | mean score: 0.06 -> 1.46 | Improved |
| add SAE reconstruction loss to projector training (ablation) | CLIP, Gemma-2-2B-it · COCO steering, replace | mean score: 0.38 -> 0.18 | Hurt |
| add SAE reconstruction loss to projector training (ablation) | I-JEPA, Gemma-2-2B-it · COCO steering, remove | mean score: 0.34 -> 0.41 | Mixed |
| use DINOv2 instead of CLIP as vision encoder (ablation) | VISTA, Gemma-2-2B-it · COCO spatial localization | spatial localization accuracy: 0.201 -> 0.927 | Improved |
| use VISTA DINOv2 steering instead of VisionSAE CLIP Global | Gemma-2-2B-it · COCO steering, remove | mean score: 1.06 -> 1.43 | Improved |
| use VISTA DINOv2 steering instead of PatchSAE | Gemma-2-2B-it · COCO steering, replace | mean score: 0.99 -> 1.46 | Improved |
| restrict VisionSAE steering to local patches | VisionSAE CLIP · COCO steering, remove | mean score: 1.06 -> 0.71 | Hurt |
| add SAE constraint to projector | DINOv2-Gemma-2-2B · LLaVA-Bench All | score: 39.2 -> 41.1 | Improved |
| add SAE constraint to projector | CLIP-Gemma-2-2B · LLaVA-Bench All | score: 38.6 -> 34.5 | Hurt |
| add SAE constraint to projector | I-JEPA-Gemma-2-2B · POPE | accuracy: 74.1 -> 73.57 | Hurt |
| add SAE constraint to projector | DINOv2-Gemma-2-2B · GQA | exact match: 43.50 -> 42.10 | Hurt |
Closest papers by meaning
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.90)
- Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.89)
- SAE-V: Interpreting Multimodal Models for Enhanced Alignment (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
