Research map / Vision-language and multimodal / CLIP and zero-shot
How Much Can CLIP Benefit Vision-and-Language Tasks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP visual encoder instead of ImageNet ResNet in task-specific VQA fine-tuning (Pythia/MCAN) | Pythia and MCAN VQA models · VQA v2.0 Test-dev | accuracy: +4.01 (Pythia), +4.26 (MCAN) for CLIP-Res50 vs ImageNet-Res50 | Helped |
| Further detection pre-training on Visual Genome applied to CLIP-Res50 encoder (ablation) | CLIP-Res50 in Pythia/MCAN (VG variants) · VQA v2.0 Test-dev | accuracy: -5.54 (PythiaVG), -4.08 (MCANVG) | Hurt |
| Use CLIP-ViT-B grid features instead of CLIP-ResNet features for VQA (ablation) | Pythia and MCAN · VQA v2.0 Test-dev | accuracy: 51.38 (Pythia) and 56.18 (MCAN) vs 65.55 and 71.49 for CLIP-Res50 | Hurt |
| Replace ImageNet encoder with CLIP-Res50 in image captioning Transformer | Transformer captioner (Luo 2020) · COCO Karpathy test | CIDEr: +9.1 CIDEr over ImageNet-Res50 | Helped |
| Use CLIP-ViT-B grid features for captioning (ablation) | Transformer captioner · COCO Karpathy test | CIDEr: 58.0 CIDEr vs 118.8 for ImageNet-Res50 | Hurt |
| Replace ImageNet ResNet features with CLIP features in EnvDrop navigation agent | EnvDrop on R2R · R2R unseen test | SR: +8 SR (59 vs 51) | Helped |
| Replace ImageNet features with CLIP features in navigation agent (CLIP-Res50 vs ImageNet-Res152) (ablation) | Attentive agent on R2R val · R2R English validation | SR: +4.4 SR (52.6 vs 48.2) | Helped |
| Zero-shot CLIP for VQA (question-answer text matching, with or without prompt engineering) (ablation) | CLIP variants zero-shot · VQA v2.0 mini-eval | accuracy: near-chance; yes/no max 0.446 for CLIP-Res50x4 PE | No clear effect |
| Unfreeze CLIP visual backbone during V&L pre-training and fine-tuning (ablation) | CLIP-Res50 in CLIP-ViLp · VQA v2.0 Test-dev | accuracy: +9.26 from pre-training (64.66 to 73.92) vs +5.72 for frozen BUTD-Res101 | Helped |
| Add V&L pre-training on image-text data to CLIP-Res50 model (ablation) | CLIP-ViLp CLIP-Res50 · VQA v2.0 Test-dev | accuracy: +9.26 (64.66 to 73.92) | Helped |
| Use CLIP-Res50 with V&L pre-training instead of frozen BUTD features in CLIP-ViLp | VQA, SNLI-VE, GQA · VQA v2.0 Test-std / SNLI-VE / GQA | accuracy: outperforms LXMERT by 2.59 on VQA | Helped |
| Use CLIP-Res50x4 visual encoder with V&L pre-training | GQA · GQA test-dev | accuracy: 61.42 vs VinVL 65.05 | Hurt |
Closest papers by meaning
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.90)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.89)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.89)
- CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation (CVPR 2025 · similarity 0.89)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.