Research map / Vision-language and multimodal / Multimodal LLMs

A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Applies a BATCH-based partial information decomposition to the predictive distributions and embeddings of 26 LVLMs on four multiple-choice VQA benchmarks to measure redundancy, unique, and synergistic information across models, layers, and training.

Problem

Accuracy-only evaluation cannot reveal whether LVLMs truly fuse vision and language or rely on unimodal (especially language) priors; existing interpretability tools are unimodal or ad hoc.

Approach

Adapts the BATCH PID estimator to LVLM multiple-choice outputs: uses mean-pooled visual and text embeddings as sources, masks one modality with calibrated Gaussian noise to get unimodal predictive distributions, and adds confidence thresholding and soft marginal aggregation, with no retraining or architectural changes.</1>

Key results

• On synergy-driven MMBench, synergy S correlates with accuracy at Spearman rho=0.750 (p<0.001) across 26 LVLMs.
• Image-removal accuracy drop correlates with S on MMBench (rho=0.809) and POPE (rho=0.744).
• Layer-wise analysis shows a consistent three-phase pattern, and during LLaVA-1.5 training synergy S rises primarily during visual instruction tuning (7B).

Surprising findings

• Within families, scaling often increases language uniqueness U2 less than synergy; in some cases U2 decreases with size, contrary to the common expectation.
• Synergy S is negligible during alignment pretraining, and for 13B LLaVA-1.5 the dominant gain during instruction tuning is in U2 rather than S.

Limitations

• PID estimation assumes a finite discrete target, so only multiple-choice VQA is covered, not open-ended generation.
• Unimodal conditionals are approximated by Gaussian-noise masking of embeddings rather than natural unimodal inputs.
• Analysis is correlational rather than causal.

Experimental setup

Datasets: MMBench POPE Reefknot PMC-VQA

Benchmarks: MMBench POPE Reefknot PMC-VQA

Metrics: PID atoms (R, U1, U2, S) accuracy Spearman correlation accuracy drop from image removal

Baselines: LLaVA-1.5 InstructBLIP Fuyu

Builds on: partial information decomposition BATCH estimator logit lens LLaVA-1.5

Closest papers by meaning

• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.93)
• Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.92)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.91)
• Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.90)
• Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain (ICML 2026 · similarity 0.90)