Research map / Vision-language and multimodal / Multimodal LLMs
Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Ground-V, a 500K-pair instruction-segmentation dataset generated with Claude 3 Sonnet from COCO images across five challenges (hallucination, multi-object, reasoning, multi-granularity, part reference), and shows it improves LISA and PSALM on referring and generalized referring segmentation benchmarks.
Problem
Current VLM-based segmentation models are unreliable under complex instructions (multi-object, reasoning, hallucinated references, multi-granular and part-level references) because existing grounding training data contains only simple, direct referring expressions.
Approach
Scale pixel-grounded instruction-following data with an automated pipeline: few-shot prompts to a teacher VLM (Claude 3 Sonnet) generate instruction-response pairs linked to existing COCO/PACO masks across five real-world challenge dimensions, with only the test set human-verified.
Key results
- Ground-V training lifts PSALM to 70.6 gIoU and 83.7 N-Acc on the gRefCOCO average, versus 50.1 gIoU and 24.5 N-Acc for PSALM without it.
- PSALM-G5 reaches 83.9/85.0/82.1 cIoU on RefCOCO val/testA/testB, the best among the compared models on most subsets.
- On the D3 unseen dataset, PSALM-G5 gains 9.0 gIoU and 23.5 N-Acc over PSALM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train PSALM with Ground-V | PSALM · gRefCOCO average | gIoU: 50.1 -> 70.6 | Helped |
| Train PSALM with Ground-V | PSALM · gRefCOCO average | N-Acc: 24.5 -> 83.7 | Helped |
| Train LISA with Ground-V | LISA · Ground-V test subsets | gIoU: 40.1 -> 53.7 | Helped |
| Train PSALM with Ground-V | PSALM · D3 | gIoU: 49.5 -> 58.5 (+9.0) | Helped |
| Train PSALM with Ground-V | PSALM · D3 | N-Acc: 42.5 -> 66.0 (+23.5) | Helped |
| Train LISA with Ground-V | LISA · ReasonSeg test | gIoU: 50.3 -> 52.1 (1.8↑) | Helped |
| Train LISA with Ground-V | LISA · MUSE test (zero-shot) | gIoU: 12.6 -> 17.5 (4.9↑) | Helped |
| Rephrase instructions using Qwen2-VL | Ground-V model · Ground-V reasoning subset | gIoU: 72.9 -> 71.5 | No clear effect |
| Increase Ground-V training data from 0% to 100% (ablation) | PSALM · Ground-V test: abstract multi-granular | gIoU: 27.3 -> 47.8 | Helped |
| Add the reasoning subset (ablation) | PSALM-G5 · Ground-V test: reasoning | gIoU: 44.1 -> 62.2 | Helped |
| Add the hallucination subset (ablation) | PSALM-G5 · Ground-V test: hallucination | N-Acc: 14.9 -> 23.3 | Helped |
| Add the hallucination subset (ablation) | PSALM-G5 · Ground-V test: fine-grained multi-granular | gIoU: 52.6 -> 51.9 | Hurt |
Surprising findings
- Training on Ground-V's hallucination subset also improves LLaVA-v1.5-7B on general VQA benchmarks (e.g., Vizwiz 50.0 to 53.6, SciQA-Img 66.8 to 69.9).
- Rephrasing the reasoning subset with Qwen2-VL leaves performance nearly unchanged (gIoU 72.9 to 71.5), suggesting limited overfitting to language style.
Limitations
- Training data contains noise because it is automatically generated by off-the-shelf models (Claude), with only the test set human-verified.
- Hallucination data can make the model overly cautious, degrading fine-grained segmentation performance on some subsets.
- Gains are reported for LISA and PSALM only; scope with other architectures is not explored.
Experimental setup
Datasets: COCO 2017 PACO RefCOCO RefCOCO+ RefCOCOg gRefCOCO ReasonSeg MUSE D3 LLaVA-v1.5-665k ADE20K COCO-Stuff
Benchmarks: RefCOCO RefCOCO+ RefCOCOg gRefCOCO D3 Ground-V test ReasonSeg MUSE POPE Vizwiz MMBench SciQA-Img
Metrics: gIoU cIoU N-Acc
Baselines: LISA PSALM GSVA GLaMM u-LLaVA PerceptionGPT PixelLM PolyFormer UNINEXT SEEM MattNet LAVT
Builds on: LISA PSALM SAM COCO 2017 PACO
Closest papers by meaning
- Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.88)
- GROUNDHOG: Grounding Large Language Models to Holistic Segmentation (CVPR 2024 · similarity 0.87)
- UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction as Reasoning (ICLR 2026 · similarity 0.87)
- VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models (CVPR 2025 · similarity 0.87)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.86)
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.