Research map / Vision-language and multimodal / Human and embodied modeling
Parrot Captions Teach CLIP to Spot Text
Yiqi Lin, Conghui He, Alex Jinpeng Wang, Bin Wang, Weijia Li, Mike Zheng Shou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train CLIP on images without embedded text instead of random LAION subset (ablation) | CLIP RN50 3M · DataComp 38 tasks | average zero-shot: 0.256 -> 0.282 | Improved |
| train CLIP only on images with embedded text (ablation) | CLIP RN50 3M · DataComp 38 tasks | average zero-shot: 0.256 -> 0.164 | Hurt |
| increase fraction of parrot captions (CoTR from 0.0 to 1.0) (ablation) | CLIP ViT-B 3M · ImageNet zero-shot | accuracy: 0.132 -> 0.013 | Hurt |
| select subsets with higher RSA (text-dominated CLIP score) (ablation) | CLIP RN50 3M · DataComp 38 tasks | average zero-shot: 0.239 -> 0.097 | Hurt |
| select subsets with higher RSC (parrot-dominated CLIP score) (ablation) | CLIP RN50 3M · DataComp 38 tasks | average zero-shot: 0.209 -> 0.084 | Hurt |
| remove spotted text from images via inpainting versus random inpainting (ablation) | OpenAI CLIP ViT-B-32 scoring · LAION-2B CLIP score | mean CLIP score: 0.3358 -> 0.2974 | Hurt |
| pre-train BLIP on images with embedded text (ablation) | BLIP 3M · TextVQA | accuracy: 15.36 -> 19.05 | Improved |
| pre-train BLIP on images with embedded text (ablation) | BLIP 3M · COCO captioning | CIDEr: 115.6 -> 108.9 | Hurt |
| filter LAION-2B by OCR (remove text images, CLIP>0.3, aesthetics>0.45, dedup) and retrain CLIP | CLIP training data · Synthetic text-spotting benchmark | sync. score: 0.368 -> 0.163 | Improved |
| debiased 100M-sample CLIP versus OpenCLIP on average zero-shot | CLIP training data · DataComp 38 tasks | average zero-shot: 0.565 -> 0.443 | Hurt |
| sample images with embedded text but no parrot caption (CoTR=0) versus random (ablation) | CLIP RN50 3M · DataComp 38 tasks | average zero-shot: 0.256 vs 0.247 | No clear effect |
Closest papers by meaning
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.88)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.88)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.87)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.87)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.87)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
