Research map / Vision-language and multimodal / Multimodal LLMs

FreeRet: MLLMs as Training-Free Retrievers

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that off-the-shelf MLLMs can serve as two-stage retrievers without training by bypassing the final MLP for embeddings, using controlled one-word summarization prompts, and reranking with a neutral multiple-choice framing.

Problem

Mixed-modality retrieval usually requires heavy contrastive post-training of MLLMs; the paper asks whether off-the-shelf MLLMs can serve as strong embedders and rerankers without any training.

Approach

Training-free two-stage retrieval from one MLLM: embeddings are taken from the last attention output (skipping the final MLP, which the authors argue induces lexicalization pressure) with controlled one-word summarization prompts; reranking uses a multiple-choice prompt that removes the LLM framing effect of Yes/No-style labels.

Key results

• FreeRet-embed with Qwen2.5-VL-7B reaches 53.7 average Precision@1 on MMEB (vs 39.8 for training-free E5-V reproduction), and FreeRet with reranking top-50 reaches 70.7, matching UniME-V2 (69.6).
• On MMEB-V2 video tasks, FreeRet with Qwen2-VL-7B reaches 63.2 video classification and 39.3 video retrieval, exceeding the listed trained baselines.
• Removing the final MLP (using h^Attn_L) improves Qwen2.5-VL 3B and 7B embeddings by 5.33 and 5.71 points on MMEB; the multiple-choice framing outperforms Yes/No by 8.4% in the framing study.

Surprising findings

• Discarding only the final MLP layer improves embeddings, while discarding additional layers degrades 7B performance.
• Right/Wrong labels yielded 5% lower accuracy than True/False, and greater label-prior bias correlated with lower accuracy (LLM framing effect).

Limitations

• Higher latency and memory than CLIP-style embedding-only retrievers, grows with number of reranking candidates.
• Performance depends on the base MLLM's representational and instruction-following quality; weak base models may degrade.
• Failure modes include missing implicit relational cues and over-reliance on visual similarity over textual instructions.

Experimental setup

Datasets: MMEB MMEB-V2 (video subset) Flickr30K MS-COCO LVBench ImageNet-1K (from MMEB, framing study)

Benchmarks: MMEB MMEB-V2 video classification MMEB-V2 video retrieval Flickr30K MS-COCO LVBench MIEB-related cross-lingual MMEB (EN/ZH/FR)

Metrics: Precision@1 Recall@K accuracy latency GPU memory

Baselines: E5-V VLM2Vec MMRet IDMR CAFe mmE5 LLaVE UniME MM-Embed LamRA GME UniME-V2

Builds on: E5-V PromptEOL MMEB

Closest papers by meaning

• MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.90)
• Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.89)
• LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.89)
• Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
• SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs (ICML 2026 · similarity 0.88)
• SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding (ICLR 2025 · similarity 0.87)