Research map / Vision-language and multimodal / Multimodal LLMs

Pathways of Visual Information Flow in Vision-Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses causal patching, attention knockouts, and corrupted-input patching on synthetic and natural visual tasks to show VLMs route visual information via a direct or text-mediated pathway, with task, data, and prompt modulating the choice and text-mediated routing acting as a fallback.

Problem

How visual information flows from image tokens to the final token in vision-language models, and whether ablation-based interventions reveal normal usage or latent alternative routes.

Approach

Distinguishes a direct pathway (last token reads image tokens) from a text-mediated pathway (image info transferred to query tokens, then read by last token) using paired causal patching that measures usage, and shows that blocking the direct pathway causes rerouting through text rather than degradation.

Key results

• Recognition restores only via image tokens (100% image-dominated on synthetic, 96.7% natural) while spatial relations are text-dominated (98% synthetic, 91.4% natural).
• Localization flips from direct on synthetic (100% image) to text-mediated on natural data (89.9% text).
• Blocking the text-mediated path r_{I->T} drops relations accuracy by up to -34.9 points on natural data, while blocking the direct path changes accuracy by at most a few points.

Surprising findings

• Corrupted-input patching recovers correct answers from text tokens alone for object recognition, even though recognition uses the direct pathway under normal inference.
• The final token does not need to attend to image tokens at all for tasks requiring visual input, since blocking that direct path has negligible effect on accuracy.

Limitations

• Limited to three single-token visual tasks and three model families with 4B-7B sizes; captioning and multi-step reasoning not studied.
• Analysis operates at the level of token groups, not specific attention heads or MLP components.
• Inference-time only; does not address how pathways emerge during training.

Experimental setup

Datasets: Shapes Recognition Shapes Localization Shapes Relations What's Up COCO Visual Genome VSR

Benchmarks: Shapes Recognition Shapes Localization Shapes Relations What's Up COCO Visual Genome VSR

Metrics: accuracy restoration score attention knockout accuracy change

Baselines: LLaVA-1.5-7B InternVL3.5-4B

Builds on: interchange interventions attention knockouts causal tracing

Closest papers by meaning

• What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.90)
• A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models (ICLR 2026 · similarity 0.90)
• Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow (CVPR 2026 · similarity 0.90)
• Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.90)
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models (ICLR 2026 · similarity 0.89)
• Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.89)