Research map / Vision-language and multimodal / Multimodal LLMs

CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CVSearch, a training-free assess-then-search framework for MLLMs that uses SAM 3 expert proposals, falling back to semantic-aware adaptive patching and bottom-up tree search for high-resolution image perception.

Problem

High-resolution image perception is a bottleneck for MLLMs; visual expert-assisted search is efficient but fails when proposals miss targets, while scan-based search is robust but computationally redundant and semantically fragmenting.

Approach

Adaptively schedule search: first check MLLM confidence in answering from the global view, then SAM 3 expert proposals, and only on expert failure run scene-aware scanning, where superpixel/agglomerative clustering on expert features creates semantically coherent patches, pruned by a visual complexity score and searched bottom-up with an adaptive sufficiency threshold.

Key results

• With InternVL2.5-8B, CVSearch raises HR-Bench 8K overall from 57.4 to 77.6 and HR-Bench 4K from 66.0 to 77.0.
• With LLaVA-OV-7B, V* Bench accuracy rises from 75.4 to 91.6, above DyFo (81.2) and DeepEyes (90.1).
• On HR-Bench 4K with Qwen2.5-VL-7B, throughput is 3.77 samples/min versus about 1.2-1.3 for ZoomEye and RAP scan-based baselines.

Surprising findings

• Pure expert-assisted SAM 3 search already reaches 84.3 on V* Bench, a large gain over the 71.2 Qwen2.5-VL-7B baseline.
• Gains persist for the strong Qwen3-VL-32B backbone (+7.5 on HR-8K), suggesting search helps even large advanced models.

Limitations

• Inference is slower than vanilla single-pass MLLM and expert-only baselines due to iterative search.
• Only SAM 3 is evaluated as the visual expert; generalization to other experts is unexplored.
• Failures occur when localization is correct but the MLLM hallucinates or misreads attributes.

Experimental setup

Datasets: V* Bench HR-Bench 4K HR-Bench 8K MME-RealWorld-Lite TreeBench FineRS-4K

Benchmarks: V* Bench HR-Bench 4K HR-Bench 8K MME-RealWorld-Lite TreeBench FineRS-4K

Metrics: accuracy throughput (samples per minute)

Baselines: SEAL DyFo ZoomEye RAP DC^2 VisCrop Pixel-Reasoner DeepEyes GPT-4o Qwen2.5-VL-32B InternVL3-38B

Builds on: ZoomEye SEAL DyFo RAP SAM 3

Closest papers by meaning

• V?: Guided Visual Search as a Core Mechanism in Multimodal LLMs (CVPR 2024 · similarity 0.92)
• BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception (ICML 2026 · similarity 0.91)
• Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models (ICML 2026 · similarity 0.90)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
• SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.89)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)