Research map / Vision-language and multimodal / Multimodal LLMs

MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMSearch-Plus, a 311-task multimodal browsing benchmark built with Spatial-Temporal Extrapolation, plus a model-agnostic search agent with Set-of-Mark zoom-in, showing the best system (o3) reaches 36-37.6% accuracy.

Problem

Existing multimodal browsing benchmarks can often be solved with text-only heuristics or a single image search, so they do not truly require fine-grained visual reasoning and long-horizon tool-augmented search.

Approach

Curate hard questions via Spatial-Temporal Extrapolation (infer off-image events, dates, venues from subtle visual cues), adversarially filter out questions answerable without search, and add a Set-of-Mark zoom-in + region-seeded image search agent framework.

Key results

• o3 with Full Rollout reaches 36.0% end-to-end accuracy on MMSearch-Plus; adding Set-of-Mark raises it to 37.6%.
• Without search, the best closed model (o3) scores 15.1%, and Qwen-2.5-VL-72B scores 0.0%.
• On the 239-task MMSearch-Plus-lite subset, o3 with text search only achieves 31.4%, the best setting there.

Surprising findings

• On easy questions, full rollout underperforms image-only search for o3 (7.4-point loss), mostly because the model skipped image search.
• o3 rarely follows a zoom-in with an image search (2.87%), while Gemini does so 25.37% of the time, and MLLMs mostly convert scene text to text queries via OCR.

Limitations

• Benchmark is small (311 tasks), and the per-category splits are correspondingly noisy.
• Benchmark is refreshed over time because newer models can answer some questions without search; results are sensitive to model internal knowledge drift.
• Set-of-Mark boxes are human-curated, so automatic mark generation is not evaluated.

Experimental setup

Datasets: MMSearch-Plus

Benchmarks: MMSearch-Plus MMSearch-Plus-lite

Metrics: accuracy

Baselines: o3 GPT-5 Gemini-2.5-Pro Qwen-2.5-VL-72B-Instruct MMSearch BrowseComp-VL MM-BrowseComp

Builds on: Set-of-Mark prompting BrowseComp MMSearch

Closest papers by meaning

• ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.90)
• OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents (NeurIPS 2026 · similarity 0.89)
• Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation (ICML 2026 · similarity 0.89)
• ReFAct: Empowering Multimodal Web Agents with Visual and Context Focusing (CVPR 2026 · similarity 0.89)
• RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.88)
• AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.88)