Research map / Vision-language and multimodal / Multimodal LLMs

Binding Visual Features Point by Point

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that learning to point-via-text in VLMs induces an internal serial visual search (identified via "search heads") that resolves binding errors and enables out-of-distribution generalization on counting and visual search.

Problem

VLMs fail on multi-object tasks due to the binding problem (binding errors from interference between compositional representations); unclear whether pointing-via-text improves performance by inducing serial visual search or for other reasons.

Approach

Combines representational analysis (RSA) and causal mediation on Molmo-7B to show pointing-via-text induces an attention-coordinate-aligned serial search routine localized in "search heads", then fine-tunes VLMs to point-to-answer to show this eliminates binding errors and generalizes OOD.

Key results

• Compositional RSA correlation r=0.9269 vs conjunctive r=0.6278 for Qwen2.5-VL-7B-Instruct, indicating compositional representations.
• In Molmo-7B, attention-centroid RMSE is lowest and causal mediation peaks at layer 19, identifying search heads.
• Point-to-answer Qwen2.5-VL generalizes to more visual search targets and to counting 11-15 objects, while direct-answer models drop to 0% on larger counts.

Surprising findings

• Direct-answer models fail to generalize even though they fit training data nearly perfectly, suggesting reliance on statistical shortcuts; on visual search, performance drops as the number of targets decreases relative to training.
• Shuffling object order during point-to-answer fine-tuning yields a model that remains robust to causal mediation but does not surpass the original point-to-answer model.

Limitations

• Fine-tuning relies on procedurally generated pointing traces, which is unlikely to scale to general-purpose tasks.
• Experiments focus on synthetic counting and visual search tasks with a limited set of models.

Experimental setup

Datasets: PixMo-Count

Benchmarks: visual search (synthetic, OOD target counts) synthetic counting (OOD object counts 11-15) real-world counting (PixMo-Count OOD 11-15)

Metrics: accuracy RMSE of attention centroids causal mediation score RSA correlation

Baselines: Direct Answer (short prompt) Direct Answer (long prompt) Points-Ans (shuffled) Points-Ans (no pts) Points-Ans (desc first) Points-Ans (dots)

Builds on: Molmo pointing representation similarity analysis causal mediation analysis chain-of-thought

Closest papers by meaning

• Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.90)
• Visual symbolic mechanisms: Emergent symbol processing in Vision Language Models (ICLR 2026 · similarity 0.89)
• Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.87)
• Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers? (NeurIPS 2025 · similarity 0.87)
• Seeing to Generalize: How Visual Data Corrects Binding Shortcuts (ICML 2026 · similarity 0.87)
• Visual Structures Help Visual Reasoning: Addressing the Binding Problem in LVLMs (NeurIPS 2025 · similarity 0.86)