Research map / Vision-language and multimodal / Multimodal LLMs

GROUNDHOG: Grounding Large Language Models to Holistic Segmentation

CVPR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 30 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GROUNDHOG, an MLLM that grounds phrases to pixel-level holistic segmentation masks by retrieving class-agnostic entity mask proposals, trained on the 2.5M-pair M3G2 instruction dataset.

Problem

Box-based grounded MLLMs ground language via location tokens, which cannot represent amorphous stuff, parts, irregular regions, or groups, and their failures are hard to diagnose.

Approach

Convert class-agnostic entity mask proposals into visual entity tokens (CLIP+DINOv2 features pooled by mask), then have <GRD> phrase query embeddings score and retrieve entities, merging selected masks into a grounding mask; mask proposals are decoupled from the MLLM so failures are diagnosable.

Key results

• On RES benchmarks, GROUNDHOG-7B reaches 78.5 cIoU on RefCOCO val and 56.2 mIoU on ReasonSeg val, outperforming LISA-7B (74.1 and 44.0).
• On POPE adversarial split, GROUNDHOG-7B reports 86.28 F1 and 86.33 accuracy, with the paper citing a 4.0% accuracy gain over the best prior model on Adversarial and 5.2% on Popular.
• On Flickr30K-Entity grounded captioning, GROUNDHOG-7B achieves CIDEr 91.3 and F1_all 32.1, versus 76.1 and 15.1 for Ferret-13B.

Measured effects reported in the paper

Surprising findings

• The original Mask2Former outperforms Mask2Former+ on RefCOCO+ (67.1 vs 66.6 in ablation), attributed to COCO-specific training.
• Using only CLIP or only DINOv2 entity features each underperforms the combined features on some datasets, but DINOv2 alone beats CLIP on RefCOCO+ (62.3 vs 59.8).

Limitations

• M3G2 relies on blended academic datasets with varying annotation quality and limited concept coverage.
• The model only processes single images; extension to 3D or video is untested.
• Performance depends on the coverage of the mask proposal model (e.g., Mask2Former+ fails to propose some exact masks).

Experimental setup

Datasets: M3G2 PNG Flickr30K-Entity RefCOCO RefCOCO+ RefCOCOg RefCLEF gRefCOCO PhraseCut D-Cube ReasonSeg RIO

Benchmarks: RefCOCO RefCOCO+ RefCOCOg gRefCOCO PhraseCut ReasonSeg RIO Flickr30K-Entities PNG TextVQA-X PointQA-Twice V7W

Metrics: cIoU mIoU Any-IoU box recall R@1 AR BLEU-4 METEOR CIDEr SPICE F1_all

Baselines: LISA PolyFormer UNINEXT-H Shikra Ferret PiGLET SAB mPLUG-Owl LLaVA MiniGPT-4 InstructBLIP MultiModal-GPT

Builds on: Mask2Former LLaVA-1.5 LISA Shikra Segment Anything

Closest papers by meaning

• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.91)
• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.90)
• PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs (ICLR 2026 · similarity 0.90)
• Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.90)
• Enhancing Part-Level Point Grounding for Any Open-Source MLLMs (CVPR 2026 · similarity 0.90)
• GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.90)