Research map / Vision-language and multimodal / Multimodal LLMs

MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MRMR, an expert-level, multidisciplinary, reasoning-intensive interleaved image-text retrieval benchmark (1,435 queries, 23 domains) with a new Contradiction Retrieval task, and benchmarks 14 retrieval models across four paradigms.

Problem

Existing multimodal retrieval benchmarks focus on general-domain, single-image, semantic-matching queries and do not test expert-level reasoning over interleaved multi-image queries and mixed-modality corpora.

Approach

Builds a benchmark from MMMU-Pro knowledge/calculation questions plus a new Contradiction Retrieval task, with human-verified positive documents and interleaved image-text queries and corpora, and evaluates four retrieval paradigms (caption+text retriever, two-stream CLIP fusion, merged-image MLLM embedders, document-as-image).

Key results

• Text retriever Qwen3-Embedding-8B with LLM-generated image captions achieves the highest average nDCG@10 of 54.1 across 11 subtasks.
• Ops-MM-Embedding reaches 67.4 on Knowledge subtasks but drops to 37.4 on Theorem and 36.6 on Contradiction tasks.
• Query expansion with Qwen2.5-VL-72B reasoning traces improves GME-Qwen2-VL by +14.8 points.

Surprising findings

• Most multimodal retrievers score below 25% Hit@1 on the synthetic Negation task, roughly equivalent to random guessing among four candidates.
• Query expansion from a weaker VLM (Qwen2-VL-2B) degrades the performance of the stronger retriever Ops-MM-Embedding.

Limitations

• Contradiction subsets are small (Negation 200 queries, Design 88, Traffic 80) and partly synthetic.
• Negative corpus sampled from PIN-14M may contain false negatives, estimated at 2.5-3.3% in audit.
• TIIR, the native interleaved model, was not publicly available and thus not evaluated.

Experimental setup

Datasets: MMMU-Pro PIN-14M BRIGHT COCO DesignQA

Benchmarks: MRMR

Metrics: nDCG@10 Hit@1

Baselines: BGE-M3 NV-Embed-v2 OpenCLIP MM-Embed GME-Qwen2-VL ColPali

Builds on: MMMU-Pro BRIGHT TIIR NegBench DesignQA

Closest papers by meaning

• RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.92)
• MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.91)
• ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.91)
• Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.91)
• MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.91)
• Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.89)