Research map / Vision-language and multimodal / Multimodal LLMs

Mechanisms of Object Localization in Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Mechanistic study of object localization and classification in LLaVA-1.5 (7B/13B) and InternVL-3.5 8B using token ablations, attention knockout, and causal mediation analysis, finding containerization of object tokens and sparse task-specific attention heads.

Problem

VLMs often fail at basic classification and localization, yet the internal mechanisms supporting object localization remain poorly understood compared to classification.

Approach

Localization is driven by a containerization mechanism where object-aligned visual tokens define the box extent regardless of their internal semantic arrangement; localization and classification rely on largely distinct, sparse attention heads, with localization depending on classification-critical heads in a sequential pipeline.

Key results

• Removing object tokens drops LLaVA-7B localization from 35.34% to 5.92% and InternVL3.5 8B localization from 72.64% to 11.27%, far more than random token removal.
• Ablating global-view object tokens in InternVL drops localization by 36.4 points versus 9.7 for local-view ablation.
• CMA finds only a handful of heads mediate effects; top-10 MF heads share only 2 heads (LLaVA) and 1 head (InternVL) between tasks.

Surprising findings

• Shuffling tokens within the object mask barely hurts localization in LLaVA-7B (35.30 vs 35.34 baseline), while full shuffling collapses it.
• Binary yes/no classification prompts show FPRs of 0.56-0.70 versus 0.27-0.45 for list-based prompts, so the list format was chosen.

Limitations

• Analysis limited to a filtered COCO subset with one object per queried category and only three models.
• CMA restricted to attention heads on fixed models; other components and training dynamics unexplored.
• Binary classification query used for CMA shows higher hallucination rate.

Experimental setup

Datasets: COCO validation (filtered subset, 2,248 annotations across 1,720 images) ImageNet validation (mean embedding and positional probe) Pascal VOC (additional evaluation)

Benchmarks: COCO validation filtered subset Pascal VOC

Metrics: IoU success rate averaged over 0.5/0.7/0.9 classification accuracy mediation fraction perplexity

Baselines: LLaVA-1.5 InternVL-3.5

Builds on: causal mediation analysis attention knockout LaMa Integrated Gradients

Closest papers by meaning

• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.92)
• Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.92)
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.90)
• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.90)
• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.90)
• Grounding Everything: Emerging Localization Properties in Vision-Language Transformers (CVPR 2024 · similarity 0.89)