Research map / Vision-language and multimodal / CLIP and zero-shot
CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes CLIP's multi-object representations with a controlled synthetic dataset (ComCO), finding the text encoder favors first-mentioned objects and the image encoder favors larger objects, and traces these biases to training data and contrastive learning.
Problem
CLIP models perform poorly in complex multi-object scenes; it is unclear how object position in captions and object size in images bias the text and image encoders' representations, and how these biases affect retrieval and image-text matching.
Approach
Introduces the ComCO dataset (Blender-rendered multi-object scenes with controlled object sizes and caption orders) to separately probe CLIP's text and image encoders via object retrieval, then links the observed biases to larger objects being mentioned first in LAION/COCO captions and to contrastive training dynamics.
Key results
- Text-based object retrieval: CLIP LAION ranks the first-mentioned object at 63.96% vs 3.76% for the fourth object on ComCO.
- Image-based object retrieval: CLIP LAION retrieves the large object at 85.45% on ComCO, versus 2.73% for the smallest object.
- Image-text matching on COCO (second scenario, where the largest object is placed last): CLIP OpenAI drops to 26.4% and NegCLIP to 28.7%, versus 63.5% and 72% in the first scenario.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Move an object later in a multi-object caption | CLIP text encoder · ComCO TOR | retrieval accuracy: 63.96 -> 3.76 | Hurt |
| Move an object later in a multi-object caption | CLIP text encoder · COCO TOR | retrieval accuracy: 67.89 -> 10.09 | Hurt |
| Use a large rather than a small object in a multi-object image | CLIP image encoder · ComCO IOR | retrieval accuracy: 85.45 vs 6.36 | Helped |
| Use a large rather than a small object in a multi-object image | CLIP image encoder · COCO IOR | retrieval accuracy: 39.44 vs 28.45 | Helped |
| Put the largest object first in the correct caption and the absent object first in the incorrect caption | CLIP models · ComCO image-text matching, first scenario | accuracy: 99.99 | Helped |
| Put the largest object last in the correct caption and the absent object last in the incorrect caption | CLIP models · ComCO image-text matching, second scenario | accuracy: 67.50 | Hurt |
| Rephrase COCO captions into the second-scenario order | CLIP openAI · COCO image-text matching | accuracy: 63.5 -> 26.4 | Hurt |
| Use a shared image-text contrastive embedding rather than a text-only sentence embedding | ComCO TOR · ComCO | retrieval accuracy: 56.28 vs 29.02; 22.71 vs 19.80; 13.17 vs 17.50; 7.48 vs 33.57 | Mixed |
| Use CLIP Datacomp rather than CLIP LAION | CLIP · ComCO TOR, first object | retrieval accuracy: 63.96 -> 71.13 | Helped |
Surprising findings
- The XLM-RoBERTa-based CLIP variant, which uses bidirectional self-attention, still shows first-mention bias, suggesting the bias does not stem from causal attention.
- SugarCrepe-finetuned CLIP shows lower text-side bias than other models, which the authors link to incomplete text representations.
Limitations
- Limited computational resources forced analysis on a random 200k-sample subset of LAION.
- The proposed mitigation is described as an initial approach specific to the current dataset.
- Text-to-image experiments and the incomplete-representation theory are only partially validated.
Experimental setup
Datasets: ComCO COCO DomainNet LAION CLEVR
Benchmarks: ComCO COCO
Metrics: top-1 retrieval accuracy image-text matching accuracy
Baselines: CLIP OpenAI CLIP LAION CLIP DataComp CLIP RoBERTa SigLIP NegCLIP SugarCrepe SBERT SimCSE
Builds on: CLIP NegCLIP SugarCrepe Sentence-BERT SimCSE
Closest papers by meaning
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.91)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.90)
- Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.90)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.89)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.89)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.