Research map / Vision-language and multimodal / CLIP and zero-shot
Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes CLIP-style models (mainly the text encoder) on concatenated image pairs with multi-positive and word-swap hard-negative captions to improve SugarCrepe++ compositionality while preserving or improving retrieval.
Problem
CLIP-style vision-language models lack compositional reasoning; prior compositionality fine-tuning methods improve lexical sensitivity (SugarCrepe) but not semantic understanding (SugarCrepe++) and often hurt retrieval and classification.
Approach
Concatenate two random images and their captions to create multiple semantically valid positives (sentence concatenations and shuffles) and a cheap hard negative by swapping same-POS words across the two scenes, combined with contrastive, single-negative, and uni-modal invariance losses plus alternating standard CLIP steps on single images.
Key results
- CLIPS + CLIC improves SugarCrepe++ average ITT to 81.0 (from 71.9 for CLIPS), the best ITT among CLIP-like models reported.
- CLIC-RedCaps on ViT-B/32 improves SugarCrepe++ replace ITT to 76.0 and swap ITT to 61.5 over CLIP's 69.5 and 45.7, while keeping ImageNet accuracy near 62.3 vs 63.3.
- CLIPS + CLIC improves COCO text retrieval by 1.3% and image retrieval by 2.2% while using only about 0.01% of pre-training samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CLIP with CLIC on RedCaps | CLIP ViT-B/32 · SugarCrepe++ Replace | ITT: 69.5 -> 76.0 | Helped |
| Fine-tune CLIP with CLIC on RedCaps | CLIP ViT-B/32 · SugarCrepe++ Swap | ITT: 45.7 -> 61.5 | Helped |
| Fine-tune CLIPS with CLIC on RedCaps | CLIPS ViT-L/14 · SugarCrepe++ Swap | ITT: 60.6 -> 75.1 | Helped |
| Fine-tune CLIPS with CLIC on RedCaps | CLIPS ViT-L/14 · MS-COCO | Image-to-text retrieval: +1.3% | Helped |
| Fine-tune CLIPS with CLIC on RedCaps | CLIPS ViT-L/14 · MS-COCO | Text-to-image retrieval: +2.2% | Helped |
| Add a single hard-negative (ablation) | CLIC concatenated-image training · SugarCrepe++ Swap | ITT: 49.3 -> 53.7 | Helped |
| Add multiple positive captions (ablation) | CLIC concatenated-image training · SugarCrepe++ Swap | ITT: 53.7 -> 59.0 | Helped |
| Add uni-modal loss (ablation) | CLIC with multiple positive captions · SugarCrepe++ Swap | ITT: 59.0 -> 59.1 | No clear effect |
| Add standard CLIP training iterations on single images (ablation) | CLIC concatenated-image training · SugarCrepe++ Swap | ITT: 59.1 -> 61.1 | Helped |
| Replace concatenated-image training with single-image CLIC (ablation) | CLIC · MS-COCO | Text retrieval: 75.9 -> 67.3 | Hurt |
| Use LLM-generated word swaps for hard negatives (ablation) | CLIC-LAION · SugarCrepe++ | ITT: 75.6 -> 76.7; 61.1 -> 60.7 | Mixed |
| Concatenate images sharing a common noun (ablation) | CLIC-LAION · Downstream tasks | performance | No clear effect |
Surprising findings
- Methods such as DAC that perform well on SugarCrepe replace ITT (89.4%) drop to 53.7% on SugarCrepe++ replace ITT, suggesting reliance on lexical cues.
- LLM-generated harder negatives improved SugarCrepe++ replace ITT but degraded swap ITT and TOT, while costing over 24 hours of API compute.
Limitations
- Only contrastive models tested; SigLIP not evaluated due to compute constraints.
- Only LLaVA-1.5 tested as large VLM; other VLMs like VILA and CogVLM left for future work.
- Relies on spaCy POS tagging and random word swaps, which can produce non-meaningful or ungrammatical negatives (about 28% meaningful by hand-labelling).
Experimental setup
Datasets: LAION RedCaps CC12M PixelProse MS-COCO ImageNet SugarCrepe SugarCrepe++ WinoGround Flickr30k CLIP_benchmark zero-shot datasets
Benchmarks: SugarCrepe++ WinoGround SugarCrepe ImageNet ZS-10 MS-COCO retrieval Flickr30k retrieval GQA TextVQA SQA-I MME
Metrics: ITT accuracy TOT accuracy Recall@5 Recall@1 zero-shot accuracy text/image/group score VQAScore
Baselines: CLIP NegCLIP DAC-LLM DAC-SAM SVLC-R SVLC-R+L CoN-CLIP TripletCLIP CLIPA CLIPS FLAVA BLIP-Base
Builds on: CLIP NegCLIP DAC SVLC TripletCLIP
Closest papers by meaning
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.89)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.89)
- Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.88)
- Learning Visual Composition through Improved Semantic Guidance (CVPR 2025 · similarity 0.88)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.87)
- Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.