Research map / Vision-language and multimodal / Multimodal LLMs
V?: Guided Visual Search as a Core Mechanism in Multimodal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEAL, an MLLM meta-architecture where an LLM-guided visual search (V*) locates missing high-resolution details into a visual working memory, and introduces V*Bench for high-resolution visual grounding.
Problem
Current MLLMs rely on low-resolution, frozen CLIP encoders and cannot identify or actively seek missing fine-grained visual details in high-resolution, visually crowded images.
Approach
An LLM-guided visual search (V*) uses an MLLM's common-sense contextual cues and target-specific heatmaps to recursively divide and search the image like A*, storing located target crops and coordinates in a visual working memory that the VQA LLM then attends to.
Key results
- SEAL achieves 75.39% overall on V*Bench (74.78% attribute, 76.31% spatial), outperforming GPT-4V (54.97%) and Gemini Pro (48.16%).
- V*-search reduces average search length to 4.65 versus 6.62 for the best sequential baseline on V*Bench.
- SEAL improves MME from 1051.2 to 1128.9 and POPE from 76.5 to 82.4 over the LLaVA* baseline, while slightly lowering MM-Vet and LLaVA-Bench scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add V* visual search and visual working memory (ablation) | LLaVA*/Vicuna-7B VQA model · V*Bench overall | accuracy: 45.02 -> 75.39 | Helped |
| Replace detector querying with V* visual search (ablation) | Vicuna-7B VQA model with visual working memory · V*Bench overall | accuracy: 62.30 -> 75.39; 62.82 -> 75.39 | Helped |
| Add the target-specific cue (ablation) | V* visual search · V*Bench | search length: 5.22 -> 4.65 | Helped |
| Add the contextual cue (ablation) | V* visual search · V*Bench | search length: 5.36 -> 4.65 | Helped |
| Add LLM-guided search | Visual search algorithm · V*Bench | search length: 6.62 -> 4.65 | Helped |
| Add LLM-guided visual search | LLaVA 7B · V*Bench | accuracy: 45.0 -> 75.3 (+30.30) | Helped |
| Add LLM-guided visual search | LLaVA 7B · MME | score: 1051.2 -> 1128.9 (+77.70) | Helped |
| Add LLM-guided visual search | LLaVA 7B · POPE | score: 76.5 -> 82.4 (+5.85) | Helped |
| Add LLM-guided visual search | LLaVA 7B · MMBench | score: 34.4 -> 33.1 (-1.36) | Hurt |
| Add LLM-guided visual search | LLaVA 7B · SEED-Bench (Img) | score: 41.8 -> 41.7 (-0.17) | No clear effect |
| Add LLM-guided visual search | LLaVA 7B · MM-Vet | score: 30.4 -> 27.7 (-2.70) | Hurt |
| Add LLM-guided visual search | LLaVA 7B · LLaVA-Bench W | score: 62.6 -> 59.1 (-3.50) | Hurt |
Surprising findings
- Off-the-shelf detectors (GroundingDINO, OWL-ViT) used in place of visual search yield much lower V*Bench accuracy (about 62%), below SEAL's 75.39%.
- The VQA LLM learning spatial relations from numeric coordinates showed a sudden 'grokking'-like loss drop after many steps.
Limitations
- Visual search model is tailored to natural images and common objects; does not transfer to document, diagram, or video without new training.
- Searching for a single target per query; exhaustive search of all instances is impractical.
- V*Bench is small (191 images, 245 targets) and uses multiple-choice options; visual search takes about 6 seconds per target on A100.
Experimental setup
Datasets: COCO2017 GQA VAW LLaVA-80K LAION-CC-SBU COCO-Stuff LVIS-PACO refCOCO refCOCO+ refCOCOg refCLEF Objects365 v2
Benchmarks: V*Bench MME POPE MMBench SEED-Bench MM-Vet LLaVA-Bench (In-the-Wild) COCO-Search18
Metrics: accuracy search length
Baselines: BLIP2 MiniGPT-4 LLaVA InstructBLIP Otter LLaVA-1.5 MM-React VisualChatGPT Visprog Bard Gemini Pro GPT-4V
Builds on: LLaVA SAM CLIP OWL-ViT A* search DETR
Closest papers by meaning
- CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception (ICML 2026 · similarity 0.92)
- InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search (ICLR 2026 · similarity 0.89)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.89)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.89)
- ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.89)
- Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.