Research map / Vision-language and multimodal / CLIP and zero-shot
Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FIxLIP, a game-theoretic method using weighted Banzhaf interaction indices with cross-modal sampling to explain image-text similarity in CLIP/SigLIP encoders via second-order token interactions, plus extended faithfulness and pointing-game metrics.
Problem
Saliency maps for vision-language encoders capture only first-order attributions and miss cross-modal interactions that drive image-text similarity; existing evaluation metrics do not handle second-order explanations.
Approach
Decompose the image-text cosine similarity into constant, token attributions and pairwise interactions via a weighted Banzhaf interaction index parameterized by mask probability p, estimated with a cross-modal sampling scheme that reuses image and text mask batches to get m_I*m_T similarity evaluations cheaply.
Key results
- On ImageNet-1k pointing game with CLIP ViT-B/32, FIxLIP (Banzhaf p=0.7) scores 0.83 for 1 object and 0.85 for 4 objects vs. 0.70 and 0.37 for Shapley values.
- FIxLIP cross-modal estimator achieves over 20x speedup in model inference time and about 5x for the full pipeline compared to the model-agnostic Shapley estimator.
- SigLIP-2 can be more faithfully explained with cross-modal interactions than CLIP at both ViT-B/32 and ViT-B/16 sizes (e.g. PGR 0.90 vs 0.83 for 1 object at ViT-B/32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use second-order FIxLIP interactions instead of first-order attributions | CLIP ViT-B/32 · ImageNet-1k pointing game with multiple objects | PGR | Helped |
| replace weighted Banzhaf interactions with Shapley interactions (ablation) | FIxLIP on CLIP ViT-B/32 · ImageNet-1k pointing game | PGR | Mixed |
| set mask weight p=0.7 rather than p=0.3 (ablation) | FIxLIP · Insertion/deletion curves, 0--50% of input deleted | faithfulness | Helped |
| set mask weight p=0.3 rather than p=0.7 (ablation) | FIxLIP · Insertion/deletion curves, 50--100% of input deleted | faithfulness | Helped |
| use cross-modal sampling instead of the model-agnostic estimator | FIxLIP · SigLIP-2 ViT-B/32 computation | speedup: over $20\times$ speedup; about $5\times$ speedup | Helped |
| use cross-modal interaction explanations rather than compare model architectures using first-order explanations | SigLIP-2 compared with CLIP · ImageNet-1k pointing game, ViT-B/32 and ViT-B/16 | PGR | Helped |
| increase encoder size from ViT-B/32 to ViT-L/16 | FIxLIP explanations · ImageNet-1k pointing game | PGR | Hurt |
Surprising findings
- SigLIP-2 is more faithfully explained by cross-modal interactions than CLIP, for both model sizes.
- Gradient-based first-order methods (Grad-ECLIP) can rank least important tokens whose deletion increases similarity, while exCLIP fails to recover nonlinear rankings.
Limitations
- Scaling to higher image resolutions and many tokens (hundreds of players) degrades faithfulness; requires two-step filtering of interactions.
- Restricted to second-order interactions; higher-order interactions and tri-modal settings are not addressed.
- Greedy subset selection is non-optimal for finding highest/lowest-sum subgraphs.
Experimental setup
Datasets: MS COCO ImageNet-1k
Benchmarks: MS COCO ImageNet-1k
Metrics: area between insertion/deletion curves (AID) pointing game recognition (PGR) p-faithfulness Spearman correlation
Baselines: GAME Grad-ECLIP exCLIP Shapley values Banzhaf values
Builds on: Shapley interaction index Faith-Banzhaf interactions KernelSHAP GAME
Closest papers by meaning
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.88)
- Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness (CVPR 2026 · similarity 0.87)
- Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.87)
- Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability (NeurIPS 2026 · similarity 0.87)
- Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.87)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.