Research map / Vision-language and multimodal / Multimodal LLMs

Seeing to Generalize: How Visual Data Corrects Binding Shortcuts

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that visual training of a small transformer (image-tokenized retrieval) shifts its text-only binding strategy from positional to symbolic, improving OOD text retrieval, and that analogous shifts appear between Qwen LLMs and their VL variants.

Problem

VLMs unexpectedly outperform their base LLMs on text-only long-context retrieval; the cause of this gap and the binding mechanisms underlying it are unclear, and text-only training yields positional shortcuts that fail out of distribution.

Approach

Visual training (image-tokenized version of the same retrieval task) removes positional shortcuts because spatial layouts lack canonical order and vision encoders are translation invariant, pushing the model to a symbolic binding mechanism that persists when text-only training resumes.

Key results

• Text-only scratch transformer gets 37.2% average OOD accuracy beyond 8 items; image-then-text model reaches 69.5%, and adding noise tokens to that model reaches 83.6%.
• Qwen3-VL-8B achieves 76.0% on a text-only retrieval task versus 62.6% for Qwen3-8B.
• Symbolic-to-positional ratio rises for every VLM over its LLM counterpart, most strongly for Qwen 3 (2.463 vs 1.819).

Surprising findings

• Training on image-tokenized versions of a text retrieval task improves text-only OOD generalization, even after text-only examples are reintroduced.
• Noise-token augmentation alone improves text-only OOD accuracy but remains below the image-trained model, suggesting visual training changes the mechanism rather than just widening positional range.

Limitations

• Mechanistic and controlled results come from small synthetic 12-layer transformers trained from scratch with a curriculum.
• Large-model analysis is limited to Qwen family interchange and probing; binding claims for other families are not tested.
• Image-trained variants with a restricted vocabulary (13 shapes) and one dropped seed run show variability across encoders and seeds.

Experimental setup

Datasets: synthetic indirect retrieval (text and rendered image)

Benchmarks: Direct Retrieval (synthetic) Indirect Retrieval (synthetic)

Metrics: accuracy standard error symbolic-to-positional ratio interchange intervention effect

Baselines: Qwen2-7B-Instruct Qwen2.5-7B-Instruct Qwen3-8B text-only scratch transformer

Builds on: interchange intervention binding mechanism taxonomy Capitals task RoPE noise token augmentation

Closest papers by meaning

• Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem (NeurIPS 2024 · similarity 0.90)
• Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.90)
• Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.90)
• Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.89)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
• Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.89)