Research map / Vision-language and multimodal / Human and embodied modeling
The Geometry of Representational Failures in Vision Language Models
Daniele Savietto, Declan Iain Campbell, André Panisson, Marco Nurisso, Giovanni Petri, Jonathan D. Cohen, Alan Perotti
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use centroid-based concept vectors instead of raw probe vectors (ablation) | activation steering of color · real-image color steering, Qwen2.5-VL 7B | steering accuracy: 3.7% -> 84.7% | Improved |
| Use centroid-based concept vectors instead of raw probe vectors (ablation) | activation steering of color · real-image color steering, Gemma 3 12B | steering accuracy: 3.7% -> 95.7% | Improved |
| Add PCA regularization to probe vectors (ablation) | probe-based concept vectors · real-image color steering, Qwen2.5-VL 7B | steering accuracy: 3.7% -> 32.2% | Improved |
| Add PCA regularization to probe vectors (ablation) | probe-based concept vectors · real-image color steering, Gemma 3 12B | steering accuracy: 3.7% -> 16.3% | Improved |
| Use PCA probe instead of centroid vectors (ablation) | concept vector extraction · real-image color steering, InternVL2.5 8B | steering accuracy: 88.4% -> 92.0% | Improved |
| Use centroid vectors instead of PCA probes for color-shape steering (ablation) | concept vector extraction · compositional steering, Qwen2.5-VL 7B | steering success: 44.2% -> 78.1% | Improved |
| Use centroid vectors instead of PCA probes for color-shape steering (ablation) | concept vector extraction · compositional steering, InternVL2.5 8B | steering success: 46.8% -> 35.9% | Hurt |
| Increase similarity of the hardest distractor to the target concept vector | visual search task · target-present trials, three VLMs | accuracy: r = -0.90 to -0.97 | Hurt |
| Increase similarity of the hardest distractor to the target concept vector | visual search task · target-absent trials, three VLMs | accuracy: r = -0.69 to -0.88 | Hurt |
| Use concept vector cosine similarity instead of HSV distance to predict model confidence (ablation) | color similarity task · three VLMs | Pearson r with logit separation: 0.60-0.66 -> 0.78-0.84 | Improved |
| Use concept vector similarity instead of HSV distance to predict chosen answer (ablation) | color similarity task · three VLMs | prediction accuracy: 70-75% for both | No clear effect |
Closest papers by meaning
- Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.92)
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.90)
- Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- Visual symbolic mechanisms: Emergent symbol processing in Vision Language Models (ICLR 2026 · similarity 0.89)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
