Research map / Vision-language and multimodal / Multimodal LLMs

UGround: Towards Unified Visual Grounding with Unrolled Transformers

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UGround, which uses RL-based stochastic selection of intermediate LLaVA transformer layers to feed a similarity-map "mask as prompt" into SAM, unifying RES, reasoning, multi-target and false-premise grounding.

Problem

Existing LMM-based visual grounding prompts SAM with the <SEG> token from the fixed last hidden layer, which accumulates layer-wise errors and lacks explicit spatial cues; prior models also cover only subsets of grounding attributes (single vs multi-target, positive vs empty target).

Approach

Policy-Prompted Masking: a REINFORCE-trained categorical policy selects which LLM layer's <SEG>-image similarity map is used as a soft logit mask prompt for SAM (skip-connection-like within a pass, dropout-like across passes), supervised by Gaussian BCE/Dice losses.

Key results

• UGround-7B reaches 63.6 gIoU / 65.4 cIoU overall on ReasonSeg test (LLaVA1.5-7B fine-tuned), up 5.4 cIoU over RSVP-GPT per the paper.
• UGround-7B achieves 76.1 on RefCOCOg test(U), +1.2 over GLaMM-7B.
• On gRefCOCO validation, UGround-LLaVA1.5-7B(ft) improves N-acc by 12.10 points over GSVA-7B (ft).

Surprising findings

• Directly using the similarity map from the last layer as a SAM prompt without training gives comparable cIoU to the trained <SEG> prompt (35.0 vs 30.7 for SESAME-type analysis).
• Training latency is higher than LISA (2.97s vs 1.26s per iteration) due to similarity-map optimization, while inference speed is comparable to LISA.

Limitations

• Predicted masks often fragmented, perforated, or spatially offset despite correct semantic grounding.
• The <SEG> token remains a language-space embedding not spatially aligned with visual tokens; SAM's internal use of mask prompts is not well understood.
• Higher training latency and memory than LISA baseline.

Experimental setup

Datasets: ReasonSeg RefCOCO RefCOCO+ RefCOCOg gRefCOCO MUSE FP-RefCOCO

Benchmarks: ReasonSeg RefCOCO RefCOCO+ RefCOCOg gRefCOCO MUSE FP-RefCOCO(+/g)

Metrics: gIoU cIoU N-acc see accuracy segment cIoU

Baselines: LISA SESAME READ GSVA PixelLM GLaMM RSVP-GPT OMG-LLaVA SegLLM VISA VideoLISA HyperSeg

Builds on: LISA LLaVA SAM READ REINFORCE

Closest papers by meaning

• VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction (CVPR 2026 · similarity 0.87)
• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.87)
• SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.87)
• Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.87)
• GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.87)
• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.86)