Research map / Vision-language and multimodal / Multimodal LLMs

Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GroundingAgent, a training-free zero-shot visual grounding pipeline that chains an LLM concept extractor, an open-vocabulary detector (YOLO-World), an MLLM region captioner, and an LLM chain-of-thought selector, reaching 65.1% average REC accuracy on RefCOCO/+/g.

Problem

Visual grounding normally requires task-specific annotations and fine-tuning, which limits generalization to novel or out-of-distribution concepts; MLLMs alone localize poorly.

Approach

Decompose grounding into agentic stages: an LLM infers candidate concepts from the query and an MLLM caption, an open-vocabulary detector proposes boxes, an MLLM describes each region, and an LLM with chain-of-thought selects the matching box from text descriptions alone, with no fine-tuning.

Key results

• GroundingAgent reaches 65.1% average zero-shot REC accuracy on RefCOCO/+/g, outperforming the best prior zero-shot method GroundVLP (62.3%).
• Replacing MLLM captions with the original query in the selection stage raises average accuracy to 90.6%, close to the pre-trained supervised SOTA (90.3%).
• Open-vocabulary detector candidate recall is around 94-98% across APE, Grounding DINO, OWL and YOLO-World.

Surprising findings

• Caption quality from the MLLM, not LLM selection, is the main bottleneck: query substitution boosts selection accuracy from 65.1% to 90.6%.
• A stronger reasoning LLM (DeepSeek-R1) gives better selection than larger-parameter non-reasoning models, showing reasoning ability matters more than size.

Limitations

• Caption hallucinations and occlusion errors from the MLLM degrade selection accuracy.
• Small referents are often missed by the detector and filtered out by the 2.5% area threshold.
• Ambiguous queries can cause all candidates to be rejected.

Experimental setup

Datasets: RefCOCO RefCOCO+ RefCOCOg

Benchmarks: RefCOCO RefCOCO+ RefCOCOg

Metrics: top-1 accuracy at IoU > 0.5 recall of candidate generation

Baselines: Pseudo-Q Grounding-DINO MM-G Kosmos-2 CoVLM GLIP REG CPT VGDiffZero ReCLIP Red Circle RelVLA

Builds on: Grounding DINO chain-of-thought SAM

Closest papers by meaning

• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.92)
• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.92)
• Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.92)
• GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.92)
• No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers (ICLR 2026 · similarity 0.91)
• SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.91)