Research map / Vision-language and multimodal / CLIP and zero-shot
MVP: Multimodality-Guided Visual Pre-training
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, Qi Tian
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace d-VAE tokenizer with CLIP vision branch as MIM target (MVP vs BEiT) | BEiT ViT-B/16 · ImageNet-1K fine-tuning | top-1: 83.2 -> 84.4 | Improved |
| replace d-VAE tokenizer with CLIP vision branch (MVP vs BEiT) | BEiT ViT-L/16 · ImageNet-1K fine-tuning | top-1: 85.2 -> 86.3 | Improved |
| replace d-VAE tokenizer with CLIP vision branch (MVP vs BEiT) | BEiT ViT-B/16 · ADE20K | mIoU: 45.6 -> 52.4 | Improved |
| replace d-VAE tokenizer with CLIP vision branch (MVP vs BEiT) | BEiT ViT-B/16 · ImageNet-1K linear probing | top-1: 37.6 -> 75.4 | Improved |
| use CLIP features as target instead of HOG (MVP vs MaskFeat) | ViT-B/16 · ImageNet-1K fine-tuning | top-1: 84.0 -> 84.4 | Improved |
| pre-train 300 epochs instead of 100 (ablation) | MVP ViT-B/16 · ImageNet-1K fine-tuning | top-1: 83.9 -> 84.4 | Improved |
| pre-train 300 epochs instead of 100 (ablation) | MVP ViT-B/16 · ADE20K | mIoU: 52.0 -> 52.4 | Improved |
| use CLIP (multimodal) guidance instead of DINO features as target (ablation) | MIM ViT-B/16 · ADE20K | mIoU: 47.0 -> 52.4 | Improved |
| use CLIP (multimodal) guidance instead of DINO features as target (ablation) | MIM ViT-B/16 · ImageNet-1K fine-tuning | top-1: 83.6 -> 84.4 | Improved |
| MVP on ImageNet-1K vs BEiT pre-trained on larger ImageNet-21K | ViT-B/16 · ADE20K | mIoU: 46.3 -> 52.4 | Improved |
| MVP on ImageNet-1K vs BEiT pre-trained on ImageNet-21K | ViT-L/16 · ADE20K | mIoU: 51.8 -> 54.3 | Improved |
| MVP linear probing vs contrastive DINO | ViT-B/16 · ImageNet-1K linear probing | top-1: 78.2 -> 75.4 | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- BEiT — BEiT: BERT Pre-Training of Image Transformers (ICLR 2022) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- An Empirical Study of Training End-to-End Vision-and-Language Transformers (CVPR 2022 · similarity 0.88)
- An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-training (IJCV 2025 · similarity 0.87)
- Mc-BEiT: Multi-Choice Discretization for Image BERT Pre-training (ECCV 2022 · similarity 0.87)
- Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight (NeurIPS 2024 · similarity 0.86)
- Instruction-Guided Visual Masking (NeurIPS 2024 · similarity 0.86)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
