Research map / Vision-language and multimodal / Multimodal LLMs

InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces O3-Bench, a high-information-density reasoning benchmark, and InSight-o3, a vReasoner/vSearcher multi-agent system whose vSearcher (InSight-o3-vS, Qwen2.5-VL-7B trained by hybrid RL) plugs into frontier models to boost performance.

Problem

Open multimodal agents lack the combined visual perception and multi-hop reasoning needed for "thinking with images" on dense charts/diagrams and maps, and existing benchmarks and visual search methods focus on single regions in natural images.

Approach

Decouple a high-level visual reasoning agent (vReasoner, a frontier model) from a visual search agent (vSearcher) trained with hybrid RL: an in-loop component using vReasoner feedback on collages and an out-of-loop component using IoU reward on pre-generated region descriptions, targeting generalized visual search for free-form region descriptions.

Key results

• GPT-5-mini on O3-Bench improves from 39.0% to 61.5% with InSight-o3-vS as vSearcher.
• Gemini-2.5-Flash on V*-Bench improves from 80.1% to 87.6% with InSight-o3-vS.
• GPT-5-nano average across benchmarks rises from 44.3% to 51.6% with InSight-o3-vS, with VisualProbe-Hard going from 21.7% to 31.4%.

Surprising findings

• Adding the untrained Qwen2.5-VL-7B as vSearcher often lowers performance, e.g. Gemini-2.5-Flash drops on VisualProbe-Hard and O3-Bench, suggesting weak tool-calling in some vReasoners.
• The average number of vSearcher tool calls stays close to 1, so the cropping tool is used less than intended.

Limitations

• Training delegates reasoning to a fixed external vReasoner (GPT-5-mini) rather than training end-to-end; joint training is left open.
• Relies on proprietary models for data generation, pseudo feedback, and evaluation, which limits reproducibility.
• O3-Bench is small (345 QA on 204 images) and results on it are highly variable across trials.

Experimental setup

Datasets: Visual CoT V* training data InfographicVQA MME-RealWorld

Benchmarks: O3-Bench V*-Bench HR-Bench 4K Tree-Bench VisualProbe-Hard MME-RealWorld Lite

Metrics: accuracy

Baselines: OpenAI o3 DeepEyes Mini-o3 Pixel Reasoner Qwen2.5-VL-7B (as vSearcher) Qwen3-VL-32B

Builds on: GRPO DeepSeek-R1 DeepEyes Mini-o3 Pixel-Reasoner

Closest papers by meaning

• Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search (ICLR 2026 · similarity 0.91)
• Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.90)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.89)
• V?: Guided Visual Search as a Core Mechanism in Multimodal LLMs (CVPR 2024 · similarity 0.89)
• Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.89)