Research map / Vision-language and multimodal / Multimodal LLMs

RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RMIR, a 1,634-query test benchmark (plus 6,687 training queries) for reasoning-intensive multimodal image retrieval across functional, temporal, and causal categories, built with an automated MLLM-based generation pipeline, and evaluates 11 retrievers.

Problem

Existing multimodal image retrieval benchmarks rely on simple direct or compositional queries and do not test retrieval that requires multi-step reasoning over an input image and text query.

Approach

An automated, scalable pipeline that generates image+text reasoning queries with MLLMs, verifies target images with a panel of MLLM judges using confidence-based filtering, and builds a retrieval pool and a test-set reliability score to limit false negatives.

Key results

• UME-R1-7B achieves the best average of 46.53% R@20 and 60.48% R@50 across functional, temporal, and causal categories.
• Generative embedding UME-R1-2B (38.09% R@20) outperforms the larger Qwen2.5-VL-7B MLLM (36.89% R@20) and VLM2Vec-Full (30.23%).
• Temporal reasoning is the hardest category; UME-R1-7B scores 40.58% R@20 on temporal versus 51.34% on functional.

Surprising findings

• Reasoning-aware training matters more than model scale: UME-R1-2B beats the larger Qwen2.5-VL-7B and VLM2Vec-7B-based models on average R@20 relative to its size.
• Qwen3-VL-2B-Instruct performs very poorly (6.77% R@20) relative to Qwen3-VL-4B (33.00%).

Limitations

• Possible false negatives remain despite multi-judge verification and the test-set reliability filter.
• Judges and query generator are from a single model family (Claude), which may share blind spots.
• Image pool is drawn only from Visual Genome and Open Images v7, limiting diversity; training data utility is not experimentally validated.

Experimental setup

Datasets: Visual Genome Open Images v7

Benchmarks: RMIR

Metrics: recall@20 recall@50

Baselines: Qwen2.5-VL-3B-Instruct Qwen2.5-VL-7B-Instruct Qwen3-VL-2B-Instruct Qwen3-VL-4B-Instruct Phi-4-Multimodal-Instruct VLM2Vec-Qwen2-VL-2B VLM2Vec-Qwen2-VL-7B VLM2Vec-Full VLM2Vec-V2 UME-R1-2B UME-R1-7B

Builds on: Self-Consistency SigLIP 2 MMEB UME-R1 Visual Genome

Closest papers by meaning

• ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.94)
• MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval (ICLR 2026 · similarity 0.92)
• MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.91)
• Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.91)
• MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoning (ICLR 2026 · similarity 0.89)
• Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.89)