Research map / Vision-language and multimodal / Multimodal LLMs

Visual symbolic mechanisms: Emergent symbol processing in Vision Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies three attention-head stages (ID retrieval, ID selection, feature retrieval) that VLMs use with content-independent spatial "position IDs" to bind features to objects, and links binding errors to failures of these mechanisms.

Problem

Vision language models fail on tasks requiring binding of visual features to objects; it is unknown whether VLMs use symbol-like, content-independent indexing mechanisms analogous to those found in text-only language models.

Approach

VLMs use visual space as a content-independent scaffold: position IDs are assigned to objects, and three sets of attention heads (ID retrieval, ID selection, feature retrieval) manipulate them. Causal mediation, representational similarity analysis, and interventions show these heads implement the mechanism, and entropy-driven binding errors trace back to degraded position IDs.

Key results

• Position-ID intervention on the PUG photorealistic task achieved over 79% efficacy across all three head types and all models.
• Binding-error intervention patching high-entropy ID-selection activations into low-entropy trials improved Qwen 2.5-VL 3B by 11.1 points (69.0% to 80.1%) and LLaVA 1.5 13B by 10.4 points.
• Ablating top-ranked CMA heads drove counting accuracy of Qwen2.5-VL-32B to 0% after 250 heads, while ablating the 500 lowest-ranked heads kept it near 70%.

Surprising findings

• Qwen-family models prefer relative over absolute spatial coding for position IDs, while LLaVA shows a weaker preference.

Limitations

• Mechanistic analysis is restricted to simplified two- and few-object synthetic scenes and a small set of open VLMs.
• Unclear whether the mechanisms are truly emergent versus driven by architectural position embeddings or training data.
• Real-world COCO ID assignment is inferred from the model's own enumeration order rather than ground truth.

Experimental setup

Datasets: synthetic colored-shape scene description images PUG photorealistic animal images COCO 2017 validation

Benchmarks: PUG-based scene description spatial reasoning task counting task COCO intervention

Metrics: intervention accuracy RSA correlation causal mediation score accuracy

Baselines: LLaVA-1.5-7B LLaVA-1.5-13B LLaVA-OneVision-7B

Builds on: binding IDs emergent symbolic mechanisms causal mediation analysis PUG

Closest papers by meaning

• Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.91)
• Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.89)
• Binding Visual Features Point by Point (NeurIPS 2026 · similarity 0.89)
• Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.89)
• Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.88)
• Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (ICLR 2026 · similarity 0.88)