Research map / Vision-language and multimodal / CLIP and zero-shot
LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-purposes a generative LMM (Qwen2-VL-7B) with LoRA modules into a universal retriever (LamRA-Ret, two-stage contrastive training) and reranker (LamRA-Rank, pointwise + listwise) for many multimodal retrieval tasks.
Problem
Multimodal information retrieval increasingly requires many complex task types (composed image, long-text, multimodal document retrieval), and existing methods rely on task-specific fine-tuning of contrastively trained VLMs such as CLIP.
Approach
Unify all retrieval tasks under one generative LMM formulation: use an explicit one-word summarization prompt to extract embeddings, train LoRA on text NLI pairs then on M-BEIR with task instructions, and add a LoRA reranker trained jointly with pointwise YES/NO and listwise position-selection objectives.
Key results
- LamRA achieves 63.7 average recall on M-BEIR versus 50.6 for UniIR-CLIP (supervised), with reranking adding about 7.1 points over LamRA-Ret (56.6).
- Zero-shot Qwen2-VL-7B averages 23.0 on M-BEIR, rising to 63.7 with the LamRA framework.
- On held-out tasks, LamRA averages 50.9 versus 44.4 for supervised UniIR-CLIP, exceeding it on InfoSeek (text-image-to-text) by 26.3 Recall@5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reranking to LamRA-Ret | LamRA · M-BEIR test set | average recall: 56.6 -> 63.7; +7.1 points | Helped |
| Add the two-stage retrieval framework to Qwen2-VL-7B | LamRA-Ret · M-BEIR test set | average recall: 23.0 -> 56.6 | Helped |
| Add language-only pre-training (ablation) | LamRA-Ret · M-BEIR test set | average recall performance: 53.6 -> 56.6 | Helped |
| Add instruction tuning (ablation) | LamRA-Ret · M-BEIR test set | average recall performance: 36.2 -> 56.6 | Helped |
| Add both pre-training and instruction tuning (ablation) | LamRA-Ret · M-BEIR test set | average recall performance: 23.0 -> 56.6 | Helped |
| Scale the base model from Qwen2-VL-2B to Qwen2-VL-7B (ablation) | LamRA-Ret · M-BEIR test set | average recall performance: 51.6 -> 56.6 | Helped |
| Scale the base model from Qwen2-VL-2B to Qwen2-VL-7B (ablation) | LamRA-Rank · M-BEIR test set | average recall performance: 58.3 -> 63.7 | Helped |
| Use pointwise reranking (ablation) | LamRA-Rank · M-BEIR test set, qt → ct | R@1: 58.2 -> 75.9 | Helped |
| Use listwise reranking (ablation) | LamRA-Rank · M-BEIR test set, qt → ct | R@1: 58.2 -> 75.9 | Helped |
| Replace pointwise reranking with listwise reranking (ablation) | LamRA-Rank · M-BEIR test set, qt → ct | R@1: 75.9 -> 75.9 | No clear effect |
Surprising findings
- Zero-shot raw Qwen2-VL-7B is weak for retrieval (23.0 average), so the adaptation stages are essential rather than the base LMM's inherent ability.
Limitations
- High inference cost of LMM-based retrieval and reranking.
- Scaling was only tested with 2B and 7B models due to compute limits.
- Video retrieval is zero-shot and falls short of InternVideo2.
Experimental setup
Datasets: NLI M-BEIR
Benchmarks: M-BEIR ShareGPT4V Urban-1K Flickr30K CIRCO GeneCIS Visual Storytelling Visual Dialog Multi-round FashionIQ CC-Neg Sugar-Crepe MSR-VTT
Metrics: Recall@K Recall@1 Accuracy MAP@5
Baselines: CLIP-L SigLIP BLIP BLIP2 Qwen2-VL-7B UniIR-BLIP UniIR-CLIP E5-V MagicLens-L EVA-CLIP-8B EVA-CLIP-18B Long-CLIP-L
Builds on: E5-V UniIR Qwen2-VL InfoNCE Explicit One-word Limitation (EOL)
Closest papers by meaning
- MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.90)
- Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.89)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.89)
- Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.89)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.