Research map / Vision-language and multimodal / Cross-modal learning

Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Synergistic Faithfulness (F_syn), a Harsanyi-dividend-based surrogate for the Shapley Interaction Index, to benchmark VLM explainers and shows unimodal faithfulness metrics produce contradictory rankings while attention-based explainers (AttnLRP, Rollout) beat VLM-native methods.

Problem

Post-hoc explainers for vision-language models are evaluated with unimodal deletion/insertion metrics, which fail under cross-modal redundancy and language priors, producing contradictory explainer rankings across modalities.

Approach

A multimodal faithfulness metric, F_syn, that computes the 2-way Harsanyi dividend between visual and textual perturbations along a K-step perturbation trajectory (6K+2 forward passes), approximating exact Shapley Interaction Index at a fraction of the cost.

Key results

• F_syn correlates with exact SII at Spearman rho=0.92 (Kendall tau=0.87) on N=200 instances, with 24x speedup (281s to 11s per instance).
• Unimodal visual vs textual SRG rankings are essentially uncorrelated globally (Kendall tau=-0.06, p=0.92).
• AttnLRP (LMM beta=0.029) and Rollout (beta=0.028) lead, while VLM-native TAM and LLaVA-CAM reach only beta=0.008 over the random baseline.

Surprising findings

• Unimodal visual deletion can rank explainers below a random baseline, and VLM-native explainers score well on visual metrics but collapse under textual perturbation.
• The best-performing attention-based methods outperform VLM-native explainers designed specifically for VLMs.

Limitations

• F_syn is more costly than unimodal metrics and unsuitable for real-time use.
• Validation restricted to VQA formats with mid-scale open-weight VLMs (2B-7B); no open-ended generation or proprietary models.
• Exact SII baseline requires aggregating features into macro-coalitions, and AttnLRP Gamma rule was omitted due to NaN instabilities.

Experimental setup

Datasets: RePOPE CVBench MMStar

Benchmarks: RePOPE CVBench MMStar

Metrics: Synergistic Faithfulness (F_syn) Spearman correlation Kendall tau LMM beta Deletion/Insertion SRG

Baselines: Random Integrated Gradients Input x Gradients GradCAM Attention Rollout Grad x Rollout AttnLRP LLaVA-CAM TAM

Builds on: Shapley Interaction Index Insertion/Deletion AUC AttnLRP LLaVA-CAM TAM

Closest papers by meaning

• Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.91)
• XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.90)
• CF-VLM：CounterFactual Vision-Language Fine-tuning (NeurIPS 2025 · similarity 0.90)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.90)
• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.90)
• Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.90)