Research map / Vision-language and multimodal / CLIP and zero-shot
VeCLIP: Improving CLIP Training via Visual-enriched Captions
Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, Meng Cao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual-enriched captions with mixed training to CLIP on web-crawled 3M | CLIP ViT-B/16 · COCO I2T retrieval (WIT-3M) | Recall@1: 5.46 -> 22.30 | Improved |
| scale pre-training data from 3M to 200M with VeCap | VeCLIP · Flickr30k I2T retrieval | Recall@1 gain over CLIP: +28.40 -> +10.20 | Mixed |
| add VeCap to CLIP at 12M for ImageNet zero-shot | CLIP ViT-B/16 · ImageNet zero-shot (WIT-12M) | Top-1 accuracy (%): 31.60 -> 38.11 | Improved |
| use VeCap captions only (no AltText) on ImageNet zero-shot (ablation) | CLIP ViT-B/16 · ImageNet zero-shot (WIT-12M) | Top-1 accuracy (%): 31.14 -> 20.99 | Hurt |
| add mixed training alternating AltText and VeCap (ablation) | VeCLIP · ImageNet zero-shot (WIT-12M) | Top-1 accuracy (%): 20.99 -> 38.11 | Improved |
| fuse LLaVA caption with AltText via LLM instead of using LLaVA caption alone (ablation) | VeCLIP · COCO I2T R@1 (WIT-12M) | Recall@1: 40.06 -> 44.52 | Improved |
| add prompt constraint in LLM rewriting (ablation) | VeCap · Flickr30k I2T R@1 (WIT-3M) | Recall@1: 37.30 -> 40.00 | Improved |
| randomly select one AltText instead of highest CLIP score (ablation) | VeCLIP mixed training · COCO I2T R@1 (WIT-3M) | Recall@1: 19.70 -> 22.30 | Improved |
| apply VeCap to well-curated CC3M captions | CLIP ViT-B/16 · COCO I2T R@1 (CC3M) | Recall@1: 13.88 -> 32.04 | Improved |
| combine VeCap with DFN data | CLIP ViT-B/16 · COCO I2T R@1 | Recall@1: 63.0 -> 66.3 | Improved |
| add VeCap to CLIP on image-to-image retrieval at 100M for Faces domain | VeCLIP · GPR1200 Faces (WIT-100M) | mAP: 51.68 -> 42.83 | Hurt |
| VeCLIP at 200M on VTAB average | CLIP ViT-B/16 · VTAB zero-shot average (WIT-200M) | Top-1 accuracy (%): 60.79 -> 62.70 | Improved |
Closest papers by meaning
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.92)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.91)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.91)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.91)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.90)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
