Research map / Large language models / Reasoning and chain of thought
Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Retrv-R1 is an R1-style reasoning MLLM for universal multimodal retrieval that reranks top-K candidates with CoT, using an information compression module with a details inspection mechanism, SFT on synthetic CoT data, and GRPO with a curriculum efficiency reward.
Problem
Applying DeepSeek-R1-style RL with chain-of-thought to MLLM-based universal multimodal retrieval is costly because many candidates consume many tokens, and naive GRPO training is unstable and yields suboptimal retrieval.
Approach
Compress each candidate into two learned tokens via an information compression module, let the model selectively request full token sequences of hard candidates during CoT (details inspection), activate reasoning with SFT on a Qwen2.5-VL-72B-synthesized retrieval CoT dataset, then run GRPO with a reward penalizing inspections whose weight is curriculum-increased over training.
Key results
- Retrv-R1-7B achieves 69.2 average Recall@5 on the M-BEIR test set versus 63.7 for LamRA-7B, and Retrv-R1-3B reaches 65.5.
- With K=50 on CIRR, Retrv-R1-7B has about 0.17x inference time and 0.29x GPU memory at K=10 relative to the Qwen2.5-VL-7B-based reference, and remains faster than LamRA-Rank-L at K=20 even at K=50.
- ICM reduces inference time by over 7x (ITR 7.40 without ICM vs 1.00 with ICM on CIRR) at a cost of about 0.9 points R@5 on CIRR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Retrv-R1-7B instead of LamRA-7B | M-BEIR retrieval · M-BEIR test set | Avg. Recall@K: 63.7 -> 69.2 | Helped |
| Use Retrv-R1-7B instead of LamRA-Rank-L-7B | CIRR retrieval, K=50 · CIRR, K=50 | R@5: 66.2 -> 72.3 | Helped |
| Use ICM token compression (ablation) | Retrv-R1-3B · CIRR | R@5: 68.8 -> 67.9 | Hurt |
| Use ICM token compression (ablation) | Retrv-R1-3B · CIRR | ITR: 7.40 -> 1.00 | Helped |
| Add the content token t_con (ablation) | Retrv-R1-3B · CIRR | R@5: 60.7 -> 67.9 | Helped |
| Add the relationship token t_rel (ablation) | Retrv-R1-3B · CIRR | R@5: 64.7 -> 67.9 | Helped |
| Add ICM self-alignment pretraining (ablation) | Retrv-R1-3B · CIRR | R@5: 64.3 -> 67.9 | Helped |
| Add the details inspection mechanism (ablation) | Retrv-R1-3B · CIRR | R@5: 62.3 -> 67.9 | Helped |
| Add the synthetic-CoT SFT stage (ablation) | Retrv-R1-3B · CIRR | R@5: 63.0 -> 67.9 | Helped |
| Add the RL stage (ablation) | Retrv-R1-3B · CIRR | R@5: 61.1 -> 67.9 | Helped |
| Replace the curriculum efficiency constraint with fixed λ = 0.5 (ablation) | Retrv-R1-3B · CIRR | R@5: 67.9 -> 66.6 | Hurt |
| Replace the curriculum efficiency constraint with fixed λ = 0.75 (ablation) | Retrv-R1-3B · CIRR | R@5: 67.9 -> 65.2 | Hurt |
Surprising findings
- Removing the ICM slightly improves accuracy (68.8 vs 67.9 on CIRR), indicating a performance-efficiency trade-off from compression.
- The number of details inspections first rises then falls during RL, reflecting an effectiveness-before-efficiency pattern under the curriculum reward.
Limitations
- The ICM causes a small accuracy drop (about 0.9% on (q^i,q^t)→c^i and 1.1% on (q^i,q^t)→(c^i,c^t)) due to compression loss.
- Relies on a 72B teacher model (Qwen2.5-VL-72B) to synthesize the CoT dataset, which is costly.
- Operates as a second-stage reranker over top-K candidates from an embedding first stage, so it depends on first-stage recall.
Experimental setup
Datasets: M-BEIR CIRR OVEN Amazon Review (Sports, Beauty, Toys) BEIR
Benchmarks: M-BEIR CIRR OVEN CIRCO GeneCIS VisDial VIST MT-FIQ OKVQA Infoseek E-VQA Amazon Review sequential recommendation
Metrics: Recall@K MAP@5 R@1 inference time ratio GPU memory ratio HR@10 NDCG@10 PR@5 VQA accuracy
Baselines: CLIP-L SigLIP BLIP BLIP2 UniIR MM-Embed LamRA Vision-R1 VLM-R1
Builds on: GRPO DeepSeek-R1 LamRA BLIP-2
Closest papers by meaning
- Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (ICLR 2026 · similarity 0.90)
- Revisual-R1: Advancing Multimodal Reasoning From Optimized Cold Start to Staged Reinforcement Learning (ICLR 2026 · similarity 0.89)
- SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs (ICML 2026 · similarity 0.88)
- Stable and Efficient Single-Rollout RL for Multimodal Reasoning (CVPR 2026 · similarity 0.88)
- Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.88)
- RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.