Research map / Vision-language and multimodal / CLIP and zero-shot
Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Interprets CLIP zero-shot image classification by measuring mutual information between textual concepts extracted from the vision encoder (visual parts described in text) and the language encoder (concepts near the predicted class), summarized by an MI-dynamics AUC across 13 CLIP models.
Problem
Existing attribution methods for CLIP zero-shot classification give coarse, entangled explanations and cannot reveal which concepts the vision and language encoders share, so the strength of shared knowledge behind a zero-shot prediction is not measured.
Approach
Training-free multimodal concepts (visual parts from spectral decomposition and PCA/K-means, each described by a textual descriptor via visual prompting and optimal transport) are mapped into a common discrete space of LLM-generated textual concepts shared with the language encoder's neighbors of the predicted class; mutual information is then computed on discrete concept indices and its ablation-
Key results
- Concept-based re-prompting with multimodal explanation descriptors improves ImageNet zero-shot top-1 accuracy by up to 3.75% (RN50x16 68.47 to 72.22).
- AUC of MI dynamics correlates with zero-shot accuracy; ViT-B/16-dfn ranks highest (AUC 4.62 PCA) and ViTs show stronger shared knowledge than CNN-based CLIP models.
- Optimal transport raises concept entropy from 1.70 to 2.33 (PCA), reducing repeated descriptors across visual concepts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multimodal descriptors to zero-shot classification | CLIP image classifiers · ImageNet validation | top-1 accuracy: +2.31 to +3.75 points for ResNets; +2.05 to +2.35 for ViTs | Helped |
| use Optimal Transport to assign diverse textual concepts (ablation) | Ours (PCA) visual concepts · ImageNet visual concepts | entropy: 1.70 -> 2.33 | Helped |
| use Optimal Transport to assign diverse textual concepts (ablation) | Ours (K-means) visual concepts · ImageNet visual concepts | entropy: 1.70 -> 2.34 | Helped |
| add FCC and CRF after nearest-neighbor interpolation (ablation) | CLIP ViT-B/32 · ImageNet validation | zero-shot accuracy: 58.09 -> 58.22 | No clear effect |
| add FCC and CRF after nearest-neighbor interpolation (ablation) | CLIP ViT-B/16 · ImageNet validation | zero-shot accuracy: 64.40 -> 64.32 | No clear effect |
| add FCC and CRF after nearest-neighbor interpolation (ablation) | CLIP ViT-L/14 · ImageNet validation | zero-shot accuracy: 72.80 -> 72.79 | No clear effect |
| add FCC and CRF after nearest-neighbor interpolation (ablation) | CLIP RN50 · ImageNet validation | zero-shot accuracy: 54.41 -> 54.43 | No clear effect |
| add FCC and CRF after nearest-neighbor interpolation (ablation) | CLIP RN101 · ImageNet validation | zero-shot accuracy: 56.83 -> 56.87 | No clear effect |
| replace GPT-3.5 with GPT-4o-mini using prompt P1 (ablation) | ViT-B/16 descriptor generation · ImageNet validation | Top-1 accuracy: 67.93 -> 68.39 | Helped |
| use prompt P4 instead of P1 with GPT-4o-mini (ablation) | ViT-B/16 descriptor generation · ImageNet validation | Top-1 accuracy: 68.39 -> 68.56 | Helped |
| use K-means instead of PCA for visual concept extraction (ablation) | multimodal explanations · ImageNet validation | Deletion: 2.460 -> 2.422 | Helped |
| use PCA instead of K-means for visual concept extraction (ablation) | multimodal explanations · ImageNet validation | Accuracy Increase: 1.781 -> 1.849 | Helped |
Surprising findings
- Higher initial MI does not imply stronger shared knowledge: the siamang class starts with higher MI than sock but has lower AUC (1.82 vs 2.10) because its curve drops faster.
Limitations
- Visual concepts from PCA/K-means are sometimes noisy and scattered, producing divergent textual descriptions.
- Analysis is restricted to zero-shot image classification; retrieval, segmentation and localization are not studied.
Experimental setup
Datasets: ImageNet Places365 Food101 WIT 400M DataComp DFN LAION-400M LAION-5B
Benchmarks: ImageNet validation Places365 Food101
Metrics: Insertion Deletion Accuracy Drop Accuracy Increase Top-1 accuracy Top-5 accuracy AUC of MI dynamics mutual information CorLoc entropy (concept diversity)
Baselines: MM-CBM MM-ProtoSim Feature Maps LF-CBM CLIP zero-shot with class descriptors (Menon & Vondrick style)
Builds on: CLIP Label-Free Concept Bottleneck Models ProtoSim Visual prompt engineering (red circle / blur) Sinkhorn-Knopp
Closest papers by meaning
- PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts (ICLR 2024 · similarity 0.92)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.91)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.90)
- Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP (ICLR 2024 · similarity 0.90)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.90)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.