Research map / Vision-language and multimodal / Human and embodied modeling
Leveraging per Image-Token Consistency for Vision-Language Pre-Training
Yunhao Gou, Tom Ko, Hansi Yang, James Kwok, Yu Zhang, Mingxuan Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EPIC to METER | METER (4M pre-training) · MSCOCO fine-tuned image retrieval | IR R@1: 59.2 -> 61.2 | Improved |
| Add EPIC to METER | METER (16M pre-training) · MSCOCO fine-tuned image retrieval | IR R@1: 59.8 -> 62.5 | Improved |
| Add EPIC to X-VLM | X-VLM (4M+ pre-training) · Flickr30K zero-shot text retrieval | TR R@1: 83.1 -> 86.2 | Improved |
| Add EPIC to METER | METER (16M+ pre-training) · NLVR2 | dev accuracy: +2.3% | Improved |
| Add EPIC to ALBEF | ALBEF (16M+ pre-training) · VQA | dev accuracy: 75.8 -> 76.7 | Improved |
| Add an ITC task with uniformly sampled inconsistent tokens (ablation) | METER · NLVR2 and retrieval tasks | downstream performance | Improved |
| Generate inconsistent tokens with a language model instead of sampling randomly (ablation) | METER with ITC · NLVR2 and retrieval tasks | downstream performance | Improved |
| Select salient image-related tokens for masking (ablation) | METER with ITC and language-model generation · NLVR2 and retrieval tasks | downstream performance | Improved |
| Generate replacements conditioned on image information (ablation) | EPIC token generator · NLVR2 and retrieval tasks | downstream performance | Hurt |
| Keep the language model fixed rather than fine-tuning it during pre-training (ablation) | EPIC token generator · Downstream tasks, especially MSCOCO fine-tuned retrieval | downstream performance | Hurt |
| Train the language model on the corpus before pre-training rather than fine-tuning it during pre-training (ablation) | EPIC token generator · NLVR2 and retrieval tasks | downstream performance | Hurt |
| Replace vanilla CMLM with saliency-based CMLM (ablation) | METER · NLVR2 and retrieval tasks | downstream performance | Improved |
Closest papers by meaning
- SaCo Loss: Sample-wise Affinity Consistency for Vision-Language Pre-training (CVPR 2024 · similarity 0.90)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.89)
- Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples (AAAI 2025 · similarity 0.89)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.89)
- SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining (CVPR 2024 · similarity 0.89)
- Efficient Vision-Language Pre-training by Cluster Masking (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
