Research map / Vision-language and multimodal / Multimodal LLMs

VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VGent, a modular encoder-decoder where a frozen MLLM (enhanced by RL-trained QuadThinker) encodes image-text hidden states and a decoder selects detector-proposed boxes via cross-attention, achieving SOTA multi-target grounding on ORES and strong REC results with constant inference latency.

Problem

Visual grounding with MLLMs is slow due to autoregressive decoding, hallucinates in multi-target scenes, and new-token methods can erode LLM reasoning; multi-target grounding remains a bottleneck.

Approach

Decouple reasoning from localization: a frozen MLLM encoder supplies multi-layer hidden states, and a decoder initialized from the LLM layers takes off-the-shelf detector proposals as queries and classifies them via cross-attention, with RL-based multi-target reasoning (QuadThinker), mask-aware IoA labels, and global target recognition added as modular upgrades.

Key results

• On ORES overall, VGent reaches 71.47 F1 vs 50.89 for RAS-13B (+20.58 F1 reported), and 68.42 gIoU / 75.28 cIoU.
• On RefCOCO/+/g, VGent averages 90.1% accuracy, +3.5 over its Qwen2.5-VL-7B backbone and above InternVL3.5-38B (89.1).
• Ablation: plugging QuadThinker-enhanced backbone into VGent raises F1 on 11+ target scenes from 15.97 (Qwen2.5-VL) to 54.53.

Surprising findings

• Jointly fine-tuning the encoder and decoder drops overall F1 from 60.55 to 45.66, suggesting unfreezing the MLLM disrupts its reasoning.
• Qwen3-VL-30B-A3B's F1 falls from 60.50 without mask-ref to 34.98 with mask-ref split.

Limitations

• Inference pipeline depends on multiple external detectors and SAM, adding total latency.
• Mask-aware labels and GRES/ReasonSeg results require additional mask-level fine-tuning.
• Oracle upper bounds show large headroom beyond current proposal selection (oracle 95.38 F1).

Experimental setup

Datasets: Objects365 MaskGroups-2M MaskGroups-HQ VisionReasoner-7K RefCOCO RefCOCO+ RefCOCOg gRefCOCO ReasonSeg

Benchmarks: ORES (MaskGroups-HQ) RefCOCO RefCOCO+ RefCOCOg gRefCOCO ReasonSeg

Metrics: F1 gIoU cIoU accuracy N-acc

Baselines: RAS Qwen3-VL-30B-A3B-Instruct Qwen2.5-VL-7B InternVL3.5-38B InternVL3.5-20B SegVG ExpVG Grounding-DINO-L UNINEXT-H ONE-PEACE Ferret-v2

Builds on: GRPO Qwen2.5-VL LISA SegVG RAS

Closest papers by meaning

• Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.91)
• No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers (ICLR 2026 · similarity 0.90)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.90)
• SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.90)
• GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
• Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.89)