Research map / Vision-language and multimodal / Multimodal LLMs
Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 1.1M-pair synthetic fused-modal (text+image) training set and trains GME, an MLLM (Qwen2-VL) contrastive embedder for universal multimodal retrieval, evaluated on a new 47-dataset UMRB benchmark.
Problem
Universal multimodal retrieval (UMR) needs a single embedder handling text, images, visual documents, and fused image-text queries/candidates, but existing MLLM-based embedders are trained on text-only data and fused-modal training data is scarce.
Approach
Synthesize large-scale fused-modal query-candidate pairs with an LLM-driven Doc2Query, entity extraction and query rewriting, and image retrieval or generation pipeline, then train an MLLM embedder with LoRA contrastive learning on a balanced mix of single-, cross-, and fused-modal data.
Key results
- GME-Qwen2VL-7B reaches 67.44 average on UMRB (47 datasets), versus 64.45 for GME-Qwen2VL-2B and 37.32 for VISTA.
- GME-Qwen2VL-2B achieves 87.84 on T→VD visual document retrieval, versus 78.21 for DSE (4.2B).
- Training on synthetic fused-modal data (GenGoogle) raises UMRB-Partial average from 51.55 (EVQA only) to 53.48 at 100K instances.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase model size from Qwen2-VL-2B to Qwen2-VL-7B | GME · UMRB T→T (16 datasets) | NDCG@10: 55.93 -> 58.19 | Helped |
| increase model size from Qwen2-VL-2B to Qwen2-VL-7B | GME · UMRB T→VD (10 datasets) | NDCG@5: 87.84 -> 89.92 | Helped |
| use LoRA rank 16 instead of rank 8 (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 59.45 -> 59.30 | Hurt |
| use LoRA rank 32 instead of rank 8 (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 59.45 -> 58.96 | Hurt |
| use LoRA rank 64 instead of rank 8 (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 59.45 -> 59.11 | Hurt |
| use full fine-tuning instead of LoRA rank 8 (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 59.45 -> 56.07 | Hurt |
| add hard-negative mining (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 58.83 -> 59.45 | Helped |
| add retrieval instructions (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 58.00 -> 59.45 | Helped |
| use EOS-token embedding instead of mean pooling (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 59.04 -> 59.45 | Helped |
| use causal attention instead of bi-directional attention (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 57.62 -> 59.45 | Helped |
| use synthesized fused-modal data instead of EVQA data (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 51.55 -> 52.97 | Helped |
| use Google-retrieved images instead of FLUX-generated images in synthetic data (ablation) | GME-Qwen2-VL-2B · UMRB-Partial | average performance: 52.97 -> 53.48 | Helped |
Surprising findings
- E5-V (8.4B, text-only trained MLLM) scores below CLIP-SF on several cross-modal and fused-modal tasks, suggesting text-only training does not transfer well to multimodal retrieval.
Limitations
- Training was halted early due to time constraints; scaling curves suggest further gains with longer training.
- Synthetic fused-modal data depends on Google Image Search or FLUX generation and on an LLM for queries, which may introduce noise; Google-retrieved images need CLIP filtering.
Experimental setup
Datasets: MS-MARCO NQ HotpotQA TriviaQA SQuAD FEVER AllNLI ImageNet LAION MSCOCO Docmatix EVQA
Benchmarks: UMRB BEIR M-BEIR ViDoRe EVQA
Metrics: NDCG@10 NDCG@5 Recall@5 Recall@10
Baselines: VISTA CLIP-SF One-Peace DSE E5-V
Builds on: E5-V Qwen2-VL ANCE Doc2Query InfoNCE
Closest papers by meaning
- MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.94)
- U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs (ICLR 2026 · similarity 0.92)
- MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval (ICLR 2026 · similarity 0.89)
- Incorporating Dense Knowledge Alignment into Unified Multimodal Representation Models (CVPR 2025 · similarity 0.89)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.89)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.