AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Parrot Captions Teach CLIP to Spot Text

Yiqi Lin, Conghui He, Alex Jinpeng Wang, Bin Wang, Weijia Li, Mike Zheng Shou

ECCV 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Concept-guided image captioning · 11 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train CLIP on images without embedded text instead of random LAION subset (ablation)CLIP RN50 3M · DataComp 38 tasksaverage zero-shot: 0.256 -> 0.282Improved
train CLIP only on images with embedded text (ablation)CLIP RN50 3M · DataComp 38 tasksaverage zero-shot: 0.256 -> 0.164Hurt
increase fraction of parrot captions (CoTR from 0.0 to 1.0) (ablation)CLIP ViT-B 3M · ImageNet zero-shotaccuracy: 0.132 -> 0.013Hurt
select subsets with higher RSA (text-dominated CLIP score) (ablation)CLIP RN50 3M · DataComp 38 tasksaverage zero-shot: 0.239 -> 0.097Hurt
select subsets with higher RSC (parrot-dominated CLIP score) (ablation)CLIP RN50 3M · DataComp 38 tasksaverage zero-shot: 0.209 -> 0.084Hurt
remove spotted text from images via inpainting versus random inpainting (ablation)OpenAI CLIP ViT-B-32 scoring · LAION-2B CLIP scoremean CLIP score: 0.3358 -> 0.2974Hurt
pre-train BLIP on images with embedded text (ablation)BLIP 3M · TextVQAaccuracy: 15.36 -> 19.05Improved
pre-train BLIP on images with embedded text (ablation)BLIP 3M · COCO captioningCIDEr: 115.6 -> 108.9Hurt
filter LAION-2B by OCR (remove text images, CLIP>0.3, aesthetics>0.45, dedup) and retrain CLIPCLIP training data · Synthetic text-spotting benchmarksync. score: 0.368 -> 0.163Improved
debiased 100M-sample CLIP versus OpenCLIP on average zero-shotCLIP training data · DataComp 38 tasksaverage zero-shot: 0.565 -> 0.443Hurt
sample images with embedded text but no parrot caption (CoTR=0) versus random (ablation)CLIP RN50 3M · DataComp 38 tasksaverage zero-shot: 0.256 vs 0.247No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy