Research map / Vision-language and multimodal / Multimodal LLMs
MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes LLaVA-Next (Mistral 7B) and NV-Embed-v1 as a universal multimodal bi-encoder with modality-aware hard negative mining, adds continuous text-retrieval fine-tuning to produce MM-Embed, and prompts MLLMs as zero-shot rerankers.
Problem
Existing retrievers handle only fixed tasks with a single modality for queries and documents, while real search needs diverse tasks over heterogeneous text, image, and interleaved text-image content; MLLM-based retrievers also show modality bias toward text in cross-modal retrieval.
Approach
Initialize the MLLM retriever from a strong text embedding LLM (NV-Embed-v1 inside LLaVA-Next), train with instruction-aware contrastive learning using modality-aware hard negatives (wrong-modality and low-ranked same-modality candidates), then continuously fine-tune on text retrieval data to recover text ability; use the MLLM's True/False probability as a zero-shot reranker over top-10 candidates
Key results
- MM-Embed achieves 52.7 average R@5 on M-BEIR global setting, the best among compared retrievers (vs 47.4 for CLIP_SF fine-tuned with random negatives).
- MM-Embed reaches 60.3 nDCG@10 on the MTEB retrieval average, ranking top-5 on the leaderboard and surpassing NV-Embed-v1 (59.36).
- Zero-shot MLLM reranking of top-10 candidates improves CIRCO mAP@5 by over 7 points over the strongest retrievers (e.g., 32.3 to 39.9 for MM-Embed).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add modality-aware hard negative mining | LLaVa-Next MLLM retrievers · M-BEIR all tasks | average retrieval accuracy: 48.3 -> 52.3 | Helped |
| continuously fine-tune M^hard(NVEmb) on text-to-text retrieval data | M^hard(NVEmb) · MTEB retrieval | average nDCG@10: 49.7 -> 60.3 | Helped |
| continuously fine-tune M^hard(NVEmb) on text-to-text retrieval data | M^hard(NVEmb) · M-BEIR all tasks | average retrieval accuracy: 52.3 -> 52.7 | Helped |
| rerank the top-10 candidates with zero-shot LLaVa-Next | MM-Embed · CIRCO validation | mAP@5: 32.3 -> 39.9 | Helped |
| rerank the top-10 candidates with zero-shot LLaVa-Next | M^hard(NVEmb) · InfoSeek task 6 | R@5: 35.8 -> 37.1 | Helped |
| rerank the top-10 candidates with zero-shot LLaVa-Next | M^hard(NVEmb) · FashionIQ task 7 | R@10: 26.6 -> 20.0 | Hurt |
| add task instructions during fine-tuning (ablation) | NVEmb · VisualNews task 1 | retrieval accuracy: 33.1 -> 38.7 | Helped |
| add task instructions to zero-shot NVEmb queries (ablation) | NVEmb · VisualNews task 1 | retrieval accuracy: 15.3 -> 17.4 | Helped |
| jointly fine-tune NVEmb on multimodal and text-to-text data (ablation) | NVEmb · M-BEIR single-modal-query tasks | M-BEIR average: 54.3 -> 52.2 | Hurt |
| jointly fine-tune NVEmb on multimodal and text-to-text data (ablation) | NVEmb · BEIR retrieval tasks | average nDCG@10: 51.7 -> 63.0 | Helped |
| continuously fine-tune M^hard(NVEmb) on text-to-text data (ablation) | M^hard(NVEmb) · M-BEIR single-modal-query tasks | M-BEIR average: 56.4 -> 55.6 | Hurt |
| continuously fine-tune M^hard(NVEmb) on text-to-text data (ablation) | M^hard(NVEmb) · BEIR retrieval tasks | average nDCG@10: 51.7 -> 63.1 | Helped |
Surprising findings
- Fine-tuning with hard negatives improves MLLM retrievers but not CLIP, and zero-shot MLLM reranking degrades accuracy on single-modal query tasks even when scaling from 7B to 34B.
- Continuous text-to-text fine-tuning on the NV-Embed-based retriever recovers text retrieval while keeping multimodal accuracy, whereas joint training degrades multimodal accuracy more.
Limitations
- MLLM-based retrievers are costlier than CLIP-based retrievers, motivating distillation.
- Zero-shot reranking hurts performance on single-modal query tasks and FashionIQ/CIRR may have unlabeled positives.
- Only text and image modalities are considered.
Experimental setup
Datasets: M-BEIR MS MARCO HotpotQA Natural Questions PAQ StackExchange SQuAD ArguAna BioASQ FiQA FEVER
Benchmarks: M-BEIR MTEB retrieval (15 tasks) CIRCO
Metrics: Recall@5 Recall@10 nDCG@10 mAP@5 modality accuracy@1
Baselines: CLIP_SF (UniIR) BLIP_FF (UniIR) MagicLens E5-V NV-Embed-v1
Builds on: NV-Embed-v1 LLaVA-NeXT UniIR / M-BEIR CLIP
Closest papers by meaning
- Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.94)
- U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs (ICLR 2026 · similarity 0.93)
- MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction (ICLR 2026 · similarity 0.92)
- MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval (ICLR 2026 · similarity 0.91)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.90)
- FreeRet: MLLMs as Training-Free Retrievers (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.