AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

Improving CLIP Training with Language Rewrites

Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, Yonglong Tian

NeurIPS 2023 · Vision-language and multimodal › CLIP and zero-shot · approach: CLIP training recipes and data curation · 35 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add LLaMA ICL caption rewrites as text augmentationCLIP ViT-B/16 · CC12M ImageNet zero-shottop-1 accuracy: 40.2 -> 48.4Improved
add LLM rewrites on LAION-400MCLIP ViT-B/32 · LAION-400M ImageNet zero-shottop-1 accuracy: 62.0 -> 64.4Improved
add LLM rewrites to SLIP (LaSLIP)SLIP · CC12M ImageNet zero-shottop-1 accuracy: 42.1 -> 49.7Improved
add LLM rewrites to CLIPCLIP · CC12M few-shot average5-way 5-shot accuracy: 83.3 -> 85.8Improved
add LLM rewrites to CLIPCLIP · CC12M ImageNet linear probeaccuracy: 70.3 -> 72.3Improved
use EDA word-level augmentation instead of LLM rewrites (ablation)LaCLIP · CC12M ImageNet zero-shottop-1 accuracy: 48.4 -> 41.2Hurt
use back translation instead of LLM rewrites (ablation)LaCLIP · CC12M ImageNet zero-shottop-1 accuracy: 48.4 -> 41.6Hurt
vary source of meta-input-output ICL examples (ChatGPT/Bard/COCO/Human) (ablation)LaCLIP · CC12M ImageNet zero-shottop-1 accuracy: 44.5 / 44.8 / 44.6 / 45.1No clear effect
increase number of LLM rewrites per caption (ablation)LaCLIP · CC12M ImageNet zero-shottop-1 accuracyImproved
use LaCLIP across ViT-S/16, B/16, L/16 (ablation)CLIP · CC12M ImageNet zero-shottop-1 accuracy: 36.9 -> 46.3 (S), 44.0 -> 49.1 (L)Improved
initialize and finetune text encoder from BERT instead of LaCLIP rewrites (ablation)CLIP · CC12M ImageNet zero-shottop-1 accuracy: 40.2 -> 42.9Improved
freeze pretrained BERT text encoder (ablation)CLIP · CC12M ImageNet zero-shottop-1 accuracy: 40.2 -> 23.2Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy