AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Leveraging per Image-Token Consistency for Vision-Language Pre-Training

Yunhao Gou, Tom Ko, Hansi Yang, James Kwok, Yu Zhang, Mingxuan Wang

CVPR 2023 · Vision-language and multimodal › Human and embodied modeling · approach: Vision-language pretraining · 9 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add EPIC to METERMETER (4M pre-training) · MSCOCO fine-tuned image retrievalIR R@1: 59.2 -> 61.2Improved
Add EPIC to METERMETER (16M pre-training) · MSCOCO fine-tuned image retrievalIR R@1: 59.8 -> 62.5Improved
Add EPIC to X-VLMX-VLM (4M+ pre-training) · Flickr30K zero-shot text retrievalTR R@1: 83.1 -> 86.2Improved
Add EPIC to METERMETER (16M+ pre-training) · NLVR2dev accuracy: +2.3%Improved
Add EPIC to ALBEFALBEF (16M+ pre-training) · VQAdev accuracy: 75.8 -> 76.7Improved
Add an ITC task with uniformly sampled inconsistent tokens (ablation)METER · NLVR2 and retrieval tasksdownstream performanceImproved
Generate inconsistent tokens with a language model instead of sampling randomly (ablation)METER with ITC · NLVR2 and retrieval tasksdownstream performanceImproved
Select salient image-related tokens for masking (ablation)METER with ITC and language-model generation · NLVR2 and retrieval tasksdownstream performanceImproved
Generate replacements conditioned on image information (ablation)EPIC token generator · NLVR2 and retrieval tasksdownstream performanceHurt
Keep the language model fixed rather than fine-tuning it during pre-training (ablation)EPIC token generator · Downstream tasks, especially MSCOCO fine-tuned retrievaldownstream performanceHurt
Train the language model on the corpus before pre-training rather than fine-tuning it during pre-training (ablation)EPIC token generator · NLVR2 and retrieval tasksdownstream performanceHurt
Replace vanilla CMLM with saliency-based CMLM (ablation)METER · NLVR2 and retrieval tasksdownstream performanceImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy