Research map / Vision-language and multimodal / CLIP and zero-shot
CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally
Darina Koishigarina, Arnas Uselis, Seong Joon Oh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hard negative captions to the training batch (ablation) | LABCLIP · PUG:SPARE (ViT-L/14) | test accuracy: 0.90 -> 0.94 | Improved |
| add hard negative captions to the training batch (ablation) | LABCLIP · PUG:SPARE (ViT-B/32) | test accuracy: 0.89 -> 0.93 | Improved |
| add hard negative captions to the training batch (ablation) | LABCLIP · ARO | accuracy | Improved |
| add a learned linear map on text embeddings | CLIP · CLEVR | cross-modal test accuracy: 0.58 -> 0.95 | Improved |
| add a learned linear map on text embeddings | CLIP · PUG:SPAR | cross-modal test accuracy: 0.53 -> 0.97 | Improved |
| add a learned linear map on text embeddings | CLIP · PUG:SPARE | cross-modal test accuracy: 0.50 -> 0.94 | Improved |
| add a learned linear map on text embeddings | CLIP ViT-B/32 · COCO retrieval | Recall@1: 0.30 -> 0.41 | Improved |
| add a learned linear map on text embeddings | CLIP ViT-B/32 · ARO COCO PRC | accuracy: 0.48 -> 0.81 | Improved |
| align the text space to the image space (cosine similarity of probe weights) | CLIP · PUG:SPARE | probe weight cosine similarity: 0.09 -> 0.65 | Improved |
| train the linear map on CC3M instead of COCO (ablation) | LABCLIP · ARO COCO PRC | accuracy: 0.81 -> 0.73 | Hurt |
| apply the linear map to image embeddings instead of text embeddings (ablation) | LABCLIP · CLEVR | cross-modal test accuracy: 0.95 -> 0.98 | No clear effect |
| use BoW-trained encoders instead of pretrained CLIP embeddings (ablation) | linear probe · CLEVR image embeddings | probe test accuracy: 0.96 -> 0.66 | Hurt |
Closest papers by meaning
- Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.90)
- CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation (CVPR 2025 · similarity 0.90)
- Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE) (NeurIPS 2024 · similarity 0.89)
- Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function (NeurIPS 2024 · similarity 0.89)
- Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.89)
- The Double-Ellipsoid Geometry of CLIP (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
