Research map / Vision-language and multimodal / Multimodal LLMs

OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Presents OpenSearch-VL, an open recipe (Wikipedia-based multi-hop VQA data, multimodal tool suite, and fatal-aware GRPO with one-sided advantage clamping) for training multimodal deep search agents on Qwen3-VL backbones.

Problem

Frontier multimodal search agents are hard to reproduce because training data, trajectory synthesis pipelines, and RL recipes are proprietary; long-horizon agentic RL also suffers from cascading tool failures and imperfect visual inputs.

Approach

Builds multi-hop image-grounded VQA from Wikipedia hyperlink paths with fuzzy entity rewriting and source-anchored grounding, equips the agent with image-enhancement and OCR tools beyond search, and trains with GRPO using fatal-aware token masking and one-sided advantage clamping so pre-failure reasoning is still learned from.

Key results

• OpenSearch-VL-30B-A3B reaches 61.6 average over seven benchmarks vs 47.8 for Qwen3-VL-30B-A3B agentic baseline.
• OpenSearch-VL-8B reaches 56.6 average, surpassing SenseNova-MARS-8B (52.7).
• OpenSearch-VL-32B reaches 63.7 average; ablation on 8B shows fatal masking with one-sided clamp improves over vanilla GRPO from 67.6 to 71.8 on three benchmarks.

Surprising findings

• Hard-masking of fatal trajectories (as in Vision-DeepResearch) gives almost no gain over vanilla GRPO (67.6 to 67.7) in the ablation.
• Removing the enhancement (image-restoration) subset causes only a small drop (64.6 to 63.3), suggesting it mainly improves robustness rather than core gains.

Limitations

• Training instability stems from external tool environment (search ranking drift, fetch failures, summarization hallucinations), inflating reward variance.
• Composite reward relies on proprietary GPT-4o and GPT-5.4 judges, which are costly and version-dependent, and score only textual queries.
• Reproducibility limited by reliance on external APIs (Serper, PaddleX OCR) and lack of multi-seed error bars.

Experimental setup

Datasets: Wikipedia hyperlink graph (2025-05-01 snapshot) LiveVQA FVQA WebQA

Benchmarks: SimpleVQA VDR MMSearch LiveVQA BrowseComp-VL FVQA InfoSeek

Metrics: Pass@1 accuracy

Baselines: GPT-4o GPT-5 Gemini-2.5-Pro Qwen3-VL-30B-A3B SenseNova-MARS-8B MMSearch-R1-7B DeepEyes-v2-7B WebWatcher-32B

Builds on: GRPO Search-R1 ReAct Vision-DeepResearch WebWatcher

Closest papers by meaning

• MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline (ICML 2026 · similarity 0.91)
• SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.90)
• Quest: Training Frontier Deep Research Agents with Fully Synthetic Tasks (NeurIPS 2026 · similarity 0.89)
• InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search (ICLR 2026 · similarity 0.89)
• ReFAct: Empowering Multimodal Web Agents with Visual and Context Focusing (CVPR 2026 · similarity 0.89)
• MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents (ICLR 2026 · similarity 0.89)