Research map / Vision-language and multimodal / Human and embodied modeling
VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMEB, a 36-dataset multimodal embedding benchmark, and VLM2Vec, a contrastive framework that turns VLMs (Phi-3.5-V, LLaVA-1.6) into instruction-following universal multimodal embedders.
Problem
Universal multimodal embedding models lack both a broad benchmark and methods; existing CLIP-style models encode modalities separately, ignore task instructions, and generalize poorly to diverse or out-of-distribution tasks.
Approach
Convert a pretrained vision-language model into an embedding model by contrastive (InfoNCE) training on instruction-prefixed query-target pairs drawn from 20 MMEB training datasets, using the last-token hidden state as the embedding, with GradCache for large batches.
Key results
- VLM2Vec (LLaVA-1.6, LoRA, 1344x1344) reaches 62.9 average Precision@1 across 36 MMEB datasets, versus 44.7 for the best non-fine-tuned baseline (UniIR CLIP_SF).
- On the 16 OOD datasets, the best VLM2Vec variant scores 57.1 versus 43.1 for the best fine-tuned baseline (OpenCLIP-FFT).
- Phi-3.5-V with LoRA reaches 60.1 overall; LoRA with rank 4 or 8 outperforms full fine-tuning (52.0) in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train VLM2Vec on MMEB instead of using the strongest non-fine-tuned baseline | LLaVA-1.6 LoRA (1344×1344) vs. UniIR (CLIP_SF) · MMEB (36 datasets) | Precision@1: 44.7 to 62.9; +18.2 | Helped |
| Train VLM2Vec on MMEB instead of the strongest non-fine-tuned baseline | LLaVA-1.6 LoRA (1344×1344) vs. UniIR (CLIP_SF) · MMEB OOD (16 datasets) | Precision@1: 41.7 to 57.1; +15.4 | Helped |
| Train VLM2Vec on MMEB instead of the strongest fine-tuned baseline | LLaVA-1.6 LoRA (1344×1344) vs. OpenCLIP-FFT · MMEB (36 datasets) | Precision@1: 47.2 to 62.9; +15.7 | Helped |
| Train VLM2Vec on MMEB instead of the strongest fine-tuned baseline | LLaVA-1.6 LoRA (1344×1344) vs. OpenCLIP-FFT · MMEB OOD (16 datasets) | Precision@1: 43.1 to 57.1; +14.0 | Helped |
| Use LoRA instead of full fine-tuning (ablation) | VLM2Vec with Phi-3.5-V · MMEB (36 datasets) | Precision@1: 52.0 to 58.4 | Helped |
| Set LoRA rank to 4 rather than full fine-tuning (ablation) | VLM2Vec with Phi-3.5-V · MMEB (36 datasets) | Precision@1: 52.0 to 58.4 | Helped |
| Set LoRA rank to 8 rather than full fine-tuning (ablation) | VLM2Vec with Phi-3.5-V · MMEB (36 datasets) | Precision@1: 52.0 to 58.2 | Helped |
| Set LoRA rank to 16 rather than full fine-tuning (ablation) | VLM2Vec with Phi-3.5-V · MMEB (36 datasets) | Precision@1: 52.0 to 50.8 | Hurt |
| Set LoRA rank to 32 rather than full fine-tuning (ablation) | VLM2Vec with Phi-3.5-V · MMEB (36 datasets) | Precision@1: 52.0 to 53.4 | Helped |
| Add task-specific instructions (ablation) | VLM2Vec with Phi-3.5-V · MMEB (36 datasets) | Precision@1: 34.8 to 52.0; +49.4% | Helped |
| Add task-specific instructions (ablation) | CLIP · MMEB (36 datasets) | Precision@1: 37.8 to 26.7; -29.4% | Hurt |
| Train on retrieval tasks rather than only classification or VQA tasks (ablation) | VLM2Vec trained on one meta-task · Unseen meta-task generalization | generalization | Helped |
Surprising findings
- Adding task instructions decreases CLIP's average score by 29.4% but improves VLM2Vec's by 49.4%.
- Full fine-tuning underperforms LoRA when the backbone is Phi-3.5-V in the rank ablation.
Limitations
- Visual grounding meta-task has only 4 datasets, limiting the meta-task generalization study.
- Performance degrades with more candidates (e.g., 1000 vs 100 candidates: 60.1 vs 76.6 on Phi-3.5-V LoRA), so scores depend on candidate-set size.
- Several OOD datasets are near-zero or weak for some baselines, and FashionIQ and CIRR remain low for VLM2Vec.
Experimental setup
Datasets: MMEB training split (20 datasets, e.g., ImageNet-1K, N24News, OK-VQA, DocVQA, MSCOCO, VisualNews, RefCOCO) M-BEIR processed versions for several retrieval datasets
Benchmarks: MMEB (36 datasets: 20 in-distribution, 16 out-of-distribution) Flickr30K
Metrics: Precision@1
Baselines: CLIP OpenCLIP SigLIP BLIP2 UniIR MagicLens E5-V CLIP-FFT OpenCLIP-FFT
Builds on: GradCache InfoNCE MTEB CLIP E5-V
Closest papers by meaning
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.91)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.91)
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.90)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.90)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.