Research map / Vision-language and multimodal / Multimodal LLMs

U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematic study of design factors (bidirectional mean pooling, progressive transition, learnable temperature with batch-size/LR scaling, filtered hard negatives, and improved reranker distillation) for MLLM-based universal multimodal retrieval, yielding U-MARVEL on M-BEIR and zero-shot tasks.

Problem

Universal multimodal retrieval with MLLM-based embedders lacks systematic understanding of which design and training choices drive performance; existing recipes differ and their mechanisms are unexplored.

Approach

Systematically ablates embedding extraction (bidirectional mean pooling vs last-token compression prompt), instruction masking, a stepwise progressive transition (NLI -> CC3M -> M-BEIR), learnable InfoNCE temperature with batch/LR scaling, filtered hard-negative mining, and an improved distillation that restricts teacher-student scoring to query, positive and top-k hard negatives, distilling a rec

Key results

• U-MARVEL single model reaches 63.2 local-pool average recall on M-BEIR, above LamRA-Ret's 56.6, and U-MARVEL+ with reranking reaches 64.8 (LamRA 63.7).
• Progressive transition, hard negative filtering (61.7 vs 60.6 local) and reranker distillation (63.2 local) each add gains in the ablation.
• Qwen3-VL-4B trained with U-MARVEL attains 58.8 local and 56.2 global average recall, exceeding LamRA-Ret (7B) at 56.6 and 54.9.

Surprising findings

• Bidirectional attention with mean pooling beats the common last-token compression-prompt approach, which diverges from GME's conclusions.
• Naively using top-k hard negatives causes training to fail completely, and increasing batch size alone yields only marginal gains without learning rate scaling.

Limitations

• Only text and image modalities are covered; audio and other modalities are left to future work.
• Experiments are limited to 7B-scale models due to submission time constraints.
• Zero-shot U-MARVEL+ (reranked) degrades on text-to-video retrieval, attributed to failure to model temporal context.

Experimental setup

Datasets: M-BEIR NLI CC3M TART ShareGPT4V

Benchmarks: M-BEIR ShareGPT4V Urban-1K CIRCO GeneCIS Flickr30K Visual Dialog Multi-round FashionIQ CC-Neg Sugar-Crepe MSR-VTT MSVD

Metrics: Recall@5 Recall@10 Average Recall MAP@5 Accuracy

Baselines: CLIP-L SigLIP UniIR-BLIP UniIR-CLIP LamRA-Ret LamRA VLM2Vec UniME E5-V MagicLens-L LLaVE-7B MM-Embed

Builds on: InfoNCE LamRA MM-Embed NV-Embed M-BEIR

Closest papers by meaning

• MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.93)
• Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.92)
• MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction (ICLR 2026 · similarity 0.91)
• NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.88)
• Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.88)
• LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.88)