AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

VeCLIP: Improving CLIP Training via Visual-enriched Captions

Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, Meng Cao

ECCV 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: CLIP training recipes and data curation · 65 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add visual-enriched captions with mixed training to CLIP on web-crawled 3MCLIP ViT-B/16 · COCO I2T retrieval (WIT-3M)Recall@1: 5.46 -> 22.30Improved
scale pre-training data from 3M to 200M with VeCapVeCLIP · Flickr30k I2T retrievalRecall@1 gain over CLIP: +28.40 -> +10.20Mixed
add VeCap to CLIP at 12M for ImageNet zero-shotCLIP ViT-B/16 · ImageNet zero-shot (WIT-12M)Top-1 accuracy (%): 31.60 -> 38.11Improved
use VeCap captions only (no AltText) on ImageNet zero-shot (ablation)CLIP ViT-B/16 · ImageNet zero-shot (WIT-12M)Top-1 accuracy (%): 31.14 -> 20.99Hurt
add mixed training alternating AltText and VeCap (ablation)VeCLIP · ImageNet zero-shot (WIT-12M)Top-1 accuracy (%): 20.99 -> 38.11Improved
fuse LLaVA caption with AltText via LLM instead of using LLaVA caption alone (ablation)VeCLIP · COCO I2T R@1 (WIT-12M)Recall@1: 40.06 -> 44.52Improved
add prompt constraint in LLM rewriting (ablation)VeCap · Flickr30k I2T R@1 (WIT-3M)Recall@1: 37.30 -> 40.00Improved
randomly select one AltText instead of highest CLIP score (ablation)VeCLIP mixed training · COCO I2T R@1 (WIT-3M)Recall@1: 19.70 -> 22.30Improved
apply VeCap to well-curated CC3M captionsCLIP ViT-B/16 · COCO I2T R@1 (CC3M)Recall@1: 13.88 -> 32.04Improved
combine VeCap with DFN dataCLIP ViT-B/16 · COCO I2T R@1Recall@1: 63.0 -> 66.3Improved
add VeCap to CLIP on image-to-image retrieval at 100M for Faces domainVeCLIP · GPR1200 Faces (WIT-100M)mAP: 51.68 -> 42.83Hurt
VeCLIP at 200M on VTAB averageCLIP ViT-B/16 · VTAB zero-shot average (WIT-200M)Top-1 accuracy (%): 60.79 -> 62.70Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy