Research map / Vision-language and multimodal / CLIP and zero-shot
Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes holistic CLIP, which replaces one-to-one image-text contrastive learning with multi-to-multi matching over (image, multi-text) data generated by VLM captioning under diverse prompts, using a multi-branch image encoder.
Problem
CLIP's one-to-one (image, text) contrastive alignment on noisy web data yields myopic models that favor short texts and shallow visual features, performing poorly on long-context and fine-grained understanding.
Approach
Generate multiple diverse captions per image with VLMs (multi-prompt or multi-VLM), encode the image into multiple embeddings via multi-branch CLS tokens or parallel MLP branches, and align them part-to-part with each text using multi-to-multi contrastive loss.
Key results
- Multi-prompt data with M2M (ΨMLP) reaches 31.2 I2T R@1 on MSCOCO vs 13.6 for CLIP trained on CC3M (CC3M setting, uni-VLM/multi-prompt).
- On CC12M, M2M with ΨCLS reaches 79.0 T2I R@1 on Sharegpt4V-5K-style long-text retrieval vs 33.9 for CLIP's I2T on Urban1K-style comparisons; Table 4 gives M2M CC12M 79.6 I2T on Urban1K vs 68.1 for OpenAI-CLIP.
- On CC12M zero-shot ImageNet top-1 rises from 39.0 (CLIP) to 48.6 (ΨCLS, multi-prompt).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace O2M with multi-to-multi contrastive learning | CLIP trained on CC12M with multi-VLM captions · Sharegpt4v-5K | I2T R@1: 61.0 -> 63.1 | Helped |
| Use multi-prompt captions instead of uni-view captions | Holistic CLIP on CC12M · ImageNet | Top-1 accuracy: 43.5 -> 48.6 | Helped |
| Use multi-to-multi learning instead of O2M learning | CC12M pre-training with uni-VLM multi-prompt captions · Fashion200-5K | I2T R@1: 13.2 -> 16.7 | Helped |
| Use holistic CLIP visual encoder instead of O2M visual encoder | ClipCap on CC12M · COCO Caption | CIDEr: 76.8 -> 82.1 | Helped |
| Use average rather than max embedding fusion (ablation) | Holistic CLIP · COCO | I2T R@1: 36.3 -> 38.2 | Helped |
| Use average rather than norm-max embedding fusion (ablation) | Holistic CLIP · COCO | I2T R@1: 36.1 -> 38.2 | Helped |
| Select main-object embeddings rather than all embeddings (ablation) | Holistic CLIP on CC12M · ImageNet | Top-1 accuracy: 48.1 -> 49.4 | Helped |
| Select long-text embeddings rather than all embeddings (ablation) | Holistic CLIP on CC12M · Sharegpt4v-5K | I2T R@1: 63.1 -> 63.5 | Helped |
| Increase caption count from 2 to 3 (ablation) | Holistic CLIP on CC3M · ImageNet | Top-1 accuracy: 29.1 -> 33.7 | Helped |
| Increase caption count from 3 to 4 (ablation) | Holistic CLIP on CC3M · ImageNet | Top-1 accuracy: 33.7 -> 38.9 | Helped |
| Increase caption count from 4 to 5 (ablation) | Holistic CLIP on CC3M · ImageNet | Top-1 accuracy: 38.9 -> 38.8 | Hurt |
| Exclude the background branch from 5 captions (ablation) | Holistic CLIP on CC3M · ImageNet | Top-1 accuracy: 38.8 -> 40.0 | Helped |
Surprising findings
- Multi-prompt captions from a single VLM can match the gains of using multiple VLMs with a uniform prompt.
- Averaging the multi-branch embeddings at inference outperforms max and norm-max fusion.
Limitations
- Multi-branch design requires training with a fixed number of branches H matched to text count M; asymmetric matching is handled by grouping or free matching.
- Caption generation relies on large VLMs, adding data construction cost and dependence on their quality.
Experimental setup
Datasets: CC3M CC12M MSCOCO Flickr30K Fashion200K ShareGPT4V Urban1K ImageNet ImageNet-A ImageNet-R
Benchmarks: MSCOCO Flickr30K Fashion200K-5K ShareGPT4V-5K Urban1K ImageNet ImageNet-A ImageNet-R COCO Caption MMBench MME SEED-Bench
Metrics: R@1 R@5 R@10 Top-1 Top-5 BLEU-4 CIDEr
Baselines: CLIP O2M contrastive learning LongCLIP ClipCap LLaVA1.5-13B
Builds on: CLIP LongCLIP one-to-multi contrastive learning
Closest papers by meaning
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.88)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.88)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.88)
- Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.88)
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.