AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

Unsupervised Vision-and-Language Pre-Training via Retrieval-Based Multi-Granular Alignment

Mingyang Zhou, Licheng Yu, Amanpreet Singh, Mengjiao Wang, Zhou Yu, Ning Zhang

CVPR 2022 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 1 citations (OpenAlex)

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use joint image-text input instead of alternating image-only and text-only input (ablation)UVLP with MLM+MRM · V+L downstream tasksmeta-average: 67.4 -> 68.1Improved
Increase the proportion of originally aligned image-text pairs (ablation)UVLP pre-training on CC · V+L downstream tasksmeta-average: 68.1 -> 73.5 -> 74.5 -> 76.0 -> 76.8 -> 77.3Improved
Add region-tag alignment to the model (ablation)μ-VLA · V+L downstream tasksmeta-average: 73.7 -> 75.0Improved
Add region-noun phrase alignment to the model (ablation)μ-VLA · V+L downstream tasksmeta-average: 69.5 -> 75.0Improved
Add image-sentence alignment to region-tag and region-noun phrase alignment (ablation)μ-VLA · V+L downstream tasksmeta-average: 75.0 -> 75.8Improved
Use image-sentence alignment rather than region-tag alignment alone (ablation)μ-VLA · NLVR2 Test-Paccuracy: 52.0 -> 71.5Improved
Use ITM alignment scores to weight retrieved-pair losses (ablation)μ-VLA · V+L downstream tasksmeta-average: 75.3 -> 75.8Improved
Retrieve weakly aligned text instead of randomly pairing text and images (ablation)μ-VLA on CC images and BookCorpus text · V+L downstream tasksmeta-average: 68.3 -> 73.8Improved
Increase retrieved candidates per image from 1 to 5 (ablation)μ-VLA using image-sentence alignment · V+L downstream tasksmeta-averageImproved
Increase retrieved candidates per image from 5 to 10 (ablation)μ-VLA using image-sentence alignment · V+L downstream tasksmeta-averageHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy