Research map / Architectures and efficiency / Attention and transformers
From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition
Francesco Gentile, Nicola Dall'Asen, Francesco Tonini, Massimiliano Mancini, Lorenzo Vaquero, Elisa Ricci
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| suppress background-related singular vectors found by SITH | OpenCLIP ViT-L/14 · Waterbirds | worst-group accuracy (%): 47.9 -> 70.6 | Improved |
| suppress singular vectors instead of zero-ablating whole heads (TextSpan) (ablation) | OpenCLIP ViT-L/14 · Waterbirds | worst-group accuracy (%): 68.0 -> 70.6 | Improved |
| randomly ablate same number of singular values (ablation) | OpenCLIP ViT-L/14 · Waterbirds | overall accuracy (%): 73.5 -> 72.0 | Hurt |
| suppress NSFW singular vectors | OpenCLIP ViT-L/14 · ViSU unsafe image query | R@10: 38.9 -> 40.4 | Improved |
| suppress NSFW singular vectors | OpenCLIP ViT-L/14 · ViSU safe text query | R@10: 75.1 -> 74.5 | No clear effect |
| rescale singular values by similarity to class names | OpenCLIP ViT-L/14 · Flowers 102 / FGVC-Aircraft / DTD | top-1 accuracy (%): 76.5 -> 77.5 (Flowers 102) | Improved |
| randomly choose singular-value scaling factors (ablation) | OpenCLIP ViT-L/14 · Flowers 102 / FGVC-Aircraft / DTD | top-1 accuracy (%): 76.5 -> 76.4 (Flowers 102) | Hurt |
| add coherence term (COMP) to NNOMP concept selection (ablation) | SITH · last layer singular vectors | interpretability vs fidelity | Improved |
| replace singular vectors with SITH reconstructions (ablation) | OpenCLIP ViT-L/14 layer 23 · ImageNet, Flowers 102, Oxford Pets, Stanford Cars | zero-shot accuracy | No clear effect |
Closest papers by meaning
- Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE) (NeurIPS 2024 · similarity 0.90)
- Interpreting CLIP with Hierarchical Sparse Autoencoders (ICML 2025 · similarity 0.88)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.88)
- Boosting the visual interpretability of CLIP via adversarial fine-tuning (ICLR 2025 · similarity 0.87)
- Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective (NeurIPS 2024 · similarity 0.87)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
