AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

Language-Image Pre-training with Long Captions

Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, Yujun Shen

ECCV 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Recaptioning and text-supervised pretraining

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
pre-train with DreamLIP long-caption objective instead of CLIP at the same data scaleCLIP ViT-B/32 · Flickr30k text retrieval, YFCC15MR@1: 34.9 -> 84.9Improved
train on 30M DreamLIP pairs versus CLIP on 400M pairsCLIP ViT-B/32 · Flickr30k text retrieval R@1R@1: 78.7 -> 87.2Improved
train on 30M DreamLIP pairs versus CLIP on 400M pairsCLIP ViT-B/32 · zero-shot ImageNettop-1 accuracy: 62.0 vs 55.7Hurt
train on 30M DreamLIP pairs versus CLIP on 400M pairsCLIP ViT-B/32 · semantic segmentation avgmIoU avg: 34.8 -> 37.3Improved
uniformly sample sub-captions from long captions as multiple positives (ablation)DreamLIP · CC3M, Flickr30k text retrievalR@1: 32.6 -> 63.0Improved
add short captions to sampled sub-captions (ablation)DreamLIP · CC3M, Flickr30k text retrievalR@1: 63.0 -> 68.3Improved
add subcaption-specific grouping loss (ablation)DreamLIP · CC3M, Flickr30k text retrievalR@1: 68.3 -> 69.5Improved
use stronger MLLM captioner (ShareGPT4V instead of InstructBLIP) (ablation)DreamLIP · CC3M, Flickr30k text retrievalR@1: 58.7 -> 69.5Improved
combine long captions from three MLLMs (ablation)DreamLIP · CC3M, Flickr30k text retrievalR@1: 69.5 -> 74.4Improved
increase sampled sub-captions K beyond about 8 (ablation)DreamLIP · CC3M, Flickr30k text retrievalR@1: 70.9 (K=8) -> 69.5 (K=10)No clear effect
increase sampled sub-captions K from 3 to 8 (ablation)DreamLIP · CC3M, Flickr30k text retrievalR@1: 65.4 -> 70.9Improved
use DreamLIP encoder in LLaVA-1.5 instead of CLIP at same scaleLLaVA-1.5 vision encoder · ScienceQA-IMG, TextVQA, POPE, MMVP avgavg score: CC3M avg 51.8 -> 52.4; ScienceQA-IMG 65.7 -> 65.1Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy