Research map / Vision-language and multimodal / Multimodal LLMs

Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces a chain-of-embedding analysis that locates a Visual Integration Point (VIP) layer in LVLMs and a Total Visual Integration (TVI) estimator that quantifies language prior strength, validated across 10 LVLMs and 6 benchmarks.

Problem

Existing analyses of language prior (LP) in LVLMs rely on input-output probing on curated datasets and cannot reveal where in the network visual information begins to influence decoding or how strong the LP is per sample.

Approach

Contrast last-token hidden states of vision-text inputs against vision-removed (blind) inputs across layers; identify a layer (VIP) where the divergence for vision-dependent samples rises, and define TVI as the mean post-VIP divergence, which serves as an inverse proxy for language prior.

Key results

• TVI computed over post-VIP layers correlates with correctness at Spearman 0.7155 on VLind-Bench and 0.6335 on ViLP for Qwen2.5-VL-7B, versus 0.0871 and -0.0364 for visual attention.
• Across 60 model-dataset settings, a visual integration point consistently appears, e.g., around layers 18-20 for Qwen2.5-VL-7B and 20-22 for Gemma-3-4B.
• Adding a TVI regularizer to LLaVA-v1.5 instruction tuning (60k subset) raised MME perception from 1369.75 to 1400.44 and reasoning from 298.21 to 321.43.

Surprising findings

• Logit-lens projected output-space divergence yields negative or weak correlation with correctness on VLind-Bench (e.g., KL -0.1693), while latent-space TVI is strong.
• Pre-VIP representation distance shows weak correlation with correctness (0.1489 for Qwen2.5-VL-7B), so only post-VIP aggregation is informative.

Limitations

• Requires white-box access to hidden states and attention, excluding closed-source APIs.
• VIP is selected manually for most models, and manually chosen VIPs are not necessarily optimal.
• Validation under realistic distribution shifts is left open; the vision-dependent/independent split is a proxy based on prediction agreement.

Experimental setup

Datasets: MME MMBench MMStar MMMU VLind-Bench ViLP CommonsenseQA COCO 2017-val llava_v1_5_mix665k

Benchmarks: MME MMBench MMStar MMMU VLind-Bench ViLP

Metrics: Spearman rank correlation accuracy p-value

Baselines: visual attention output divergence PAI

Builds on: chain-of-embedding logit lens PAI

Closest papers by meaning

• A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models (ICLR 2026 · similarity 0.92)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.92)
• Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.91)
• Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.91)
• Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.91)