Research map / Vision-language and multimodal / Multimodal LLMs

Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View

ICML 2024 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Harsanyi-dividend-based interaction metrics to diagnose whether the text encoders, image encoders, and their cross-modal matching of five VLMs encode compositional (object, relation, attribute) knowledge, concluding text encoders are strong, image encoders weaker, and their knowledge is mutually unmatched.

Problem

Black-box accuracy benchmarks show VLMs fail at compositional reasoning but do not explain the internal cause; the paper asks whether the text encoder, image encoder, or their cross-modal alignment of compositional knowledge is responsible.

Approach

Treats VLM matching score as a game over image-region and text-token players and uses Harsanyi dividends (interaction effects) to define sensitivity metrics for relation, object, attribute and their interactions, applied separately to text encoders, image encoders, and cross-modal pairs.

Key results

• On Visual Genome Relation, Q_{R&O} is the largest sensitivity for all five text encoders (e.g., CLIP 1.3e-2 vs Q_O 4.5e-3 and Q_R 9.8e-6).
• On EQBEN image encoders, the relation-within-pair metric D_{O1&O2} is not larger than per-object metrics for most VLMs (e.g., CLIP D_{O1&O2}=2.0e-2 vs D_{O2}=3.1e-2), indicating weaker visual compositional sensitivity.
• Cross-modal matching fails: for CLIP, D_{I:O1&O2->T:O}=2.4e0 exceeds D_{I:O1&O2->T:R&O}=1.3e0, so text and image encoders do not share matching compositional knowledge.

Measured effects reported in the paper

Surprising findings

• Text encoders recognize relation-object interaction differences well despite poor benchmark accuracy.
• Image encoders are less sensitive to relation changes between objects than to changes in each object alone.

Limitations

• Image-side analysis restricted to 290 EQBEN pairs with SAM-derived masks, with manual correction.
• Analysis limited to object pairs with differing relations; more complex visual cases left for future work.
• Word-role segmentation via spaCy part-of-speech tagging is rough.

Experimental setup

Datasets: Visual Genome Relation Visual Genome Attribution EQBEN VL-CheckList SUGARCREPE

Benchmarks: ARO EQBEN SUGARCREPE VL-CheckList

Metrics: Harsanyi-dividend sensitivity metrics Q and D Pearson correlation between reward and interaction differenc

Baselines: CLIP NegCLIP BLIP XVLM FLAVA

Builds on: Harsanyi dividend Shapley value ARO SUGARCREPE

Closest papers by meaning

• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.91)
• ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs (NeurIPS 2024 · similarity 0.90)
• Iterated Learning Improves Compositionality in Large Vision-Language Models (CVPR 2024 · similarity 0.88)
• Are Large Vision Language Models Good Game Players? (ICLR 2025 · similarity 0.88)
• Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.88)
• Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.88)