Research map / Vision-language and multimodal / CLIP and zero-shot
Learning Visual Composition through Improved Semantic Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Improves CLIP's compositional understanding by recaptioning WebLI images with Gemini 1.5 Flash, swapping in a pretrained LLM text encoder, and adding synthetic hard negatives, achieving SOTA on ARO, SugarCrepe, and DOCCI retrieval without architectural changes.
Problem
Contrastive vision-language models such as CLIP act like bags of words and fail at capturing attributes and relations between objects, with weak image-caption retrieval on detailed captions.
Approach
Keep a standard two-tower contrastive CLIP and instead enrich the supervision: recaption web images with a multimodal foundation model grounded on alt-text, replace the text tower with a pretrained LLM encoder (partially unfrozen), and add synthetic hard negative captions.
Key results
- Recaptioning plus Gemini text encoder raises DOCCI-full recall@1 from 46.2 for CLIP to 89.7 for the proposed model, and 94.5 on DOCCI test split.
- ARO accuracy reaches 92% on relations and 94% on attributes, outperforming prior bespoke architectures such as X-VLM (73%, 87%).
- Zero-shot ImageNet top-1 reaches 68.4%, rising to 79.1% with a 50% JFT mixture, below CoCa's 82.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use grounded recaptioning and a pretrained text encoder instead of standard CLIP supervision | CLIP · DOCCI-test | recall @ 1: 58.4% to 94.5% | Helped |
| Use grounded recaptioning and a pretrained text encoder instead of standard CLIP supervision | CLIP · ARO relations | macro accuracy: 59% -> 92% | Helped |
| Use grounded recaptioning and a pretrained text encoder instead of standard CLIP supervision | CLIP · ARO attributes | macro accuracy: 63% -> 94% | Helped |
| Use grounded recaptioning and a pretrained text encoder instead of standard CLIP supervision | CLIP · ImageNet | zero-shot top-1 accuracy: 76.2 to 68.4 | Hurt |
| Add a 50% mixture of JFT to training (ablation) | Ours · ImageNet | zero-shot top-1 accuracy: 68.4% to 79.1% | Helped |
| Replace alt-text with recaptioned training data (ablation) | CLIP with scratch text encoder · DOCCI-full | recall @ 1: 53.5 to 75.6 | Helped |
| Use a Gemini-8B text encoder instead of a scratch text encoder with alt-text (ablation) | CLIP · DOCCI-full | recall @ 1: 53.5 to 67.2 | Helped |
| Use a Gemini-8B text encoder with recaptioned data instead of a scratch text encoder with recaptioned data (ablation) | CLIP · DOCCI-full | recall @ 1: 75.6 to 91.6 | Helped |
| Use Gemma-2B instead of Gemini-8B as the text encoder with recaptioned data (ablation) | CLIP · DOCCI-full | recall @ 1: 91.6 to 88.9 | Hurt |
| Remove alt-text grounding from recaptioning (ablation) | CLIP with Gemini 1.5 Flash captions · DOCCI-full | recall @ 1: 90.3 to 89.3 | No clear effect |
| Use short captions instead of default-length captions (ablation) | CLIP with Gemini 1.5 Flash captions · DOCCI-full | recall @ 1: 90.3 to 85.2 | Hurt |
| Add random sentence sampling (ablation) | CLIP with recaptioned data · COCO | recall @ 1: 51.9 to 56.3 | Helped |
Surprising findings
- Standard CLIP caption-based image retrieval on COCO is limited by noisy COCO captions: 70.2% of 'failures' judged acceptable matches by human raters.
- Text-only Gemini and Vera reach strong ARO scores, showing some captions can be solved without the image.
Limitations
- Relies on proprietary Gemini models for recaptioning and text encoding, limiting reproducibility.
- Lower zero-shot ImageNet accuracy (68.4%) than CoCa and CLIP; ImageNet and COCO retrieval evaluations are noisy or misaligned.
- Evaluated at reduced resolution (256x256) and without fine-tuning on DOCCI train split.
Experimental setup
Datasets: WebLI (English subset, 1B images) CC-12M DOCCI COCO ARO SugarCrepe ImageNet JFT
Benchmarks: ARO SugarCrepe DOCCI COCO ImageNet
Metrics: accuracy recall@1 top-1 accuracy macro accuracy
Baselines: CLIP CoCa NegCLIP BLIP X-VLM VeCLIP Long-CLIP MATE TIPS Vera GPT-4V
Builds on: CLIP NegCLIP CoCa Long-CLIP
Closest papers by meaning
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.89)
- Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.88)
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.88)
- Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Progressive Compositionality in Text-to-Image Generative Models (ICLR 2025 · similarity 0.88)
- Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.