Research map / Vision-language and multimodal / CLIP and zero-shot
GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GroundVLP, a training-free zero-shot visual grounding method that fuses GradCAM heatmaps from image-text matching VLP models with candidate boxes from an open-vocabulary detector (Detic) via a weighted grade.
Problem
Visual grounding (referring expression comprehension and phrase grounding) is limited by scarce task-specific annotations, so models generalize poorly; the paper asks whether grounding can be done zero-shot using image-text matching VLP models and open-vocabulary detectors.
Approach
Combine a GradCAM heatmap from an ITM head of a VLP model (aggregated only over visually recognizable words) with category-filtered boxes from an open-vocabulary detector, selecting the box with the highest weighted grade (detector score times heatmap mass divided by area^alpha).
Key results
- GroundVLP w/ VinVL with ground-truth category reaches 73.50% on RefCOCO testA and 55.01% on testB, versus ReCLIP w/ relations at 46.10% and 47.07%.
- On Flickr30k test, GroundVLP w/ VinVL achieves R@1 64.19%, compared with 27.57% for CPT-adapted.
- The weighted grade fusion improves over raw heatmap selection by about 3 points on RefCOCO with ALBEF (55.13 to 58.22 with theta=0.15 and s_k weighting).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the ground-truth category instead of the predicted category | GroundVLP w/ VinVL · RefCOCO testA | accuracy (%): 69.21 -> 73.50 | Helped |
| add detector confidence s_k to the grade (ablation) | GroundVLP w/ ALBEF · RefCOCO val | accuracy (%): 55.13 -> 58.22 | Helped |
| raise the detector threshold from 0.15 to 0.50 while using s_k (ablation) | GroundVLP w/ ALBEF · Flickr30k val | accuracy (%): 63.76 -> 62.12 | Hurt |
| restrict candidates to the predicted category rather than using all proposals (ablation) | GroundVLP w/ ALBEF · RefCOCOg val | accuracy (%): 55.11 -> 64.30 | Helped |
| use visual-word attention aggregation instead of averaging all text tokens (ablation) | GroundVLP w/ ALBEF · RefCOCO val | accuracy (%): 57.34 -> 58.22 | Helped |
| replace GroundVLP with Detic-only grounding | GroundVLP w/ ALBEF · RefCOCO val | accuracy (%): 58.22 -> 6.65 | Hurt |
| fine-tune ALBEF on RefCOCO+ training data | GroundVLP w/ ALBEF · RefCOCO+ val | accuracy (%): 63.57 -> 68.40 | Helped |
| use visual-word attention aggregation (ablation) | GroundVLP w/ VinVL · RefCOCO val | accuracy (%): 65.01 -> 65.01 | No clear effect |
Surprising findings
- Detic alone with the ground-truth category and query-embedding similarity gets only about 6.7% on RefCOCO/+ and 8.5% on RefCOCOg, showing OVDs lack grounding ability on their own.
- Predicted category extraction causes a notable accuracy drop versus ground-truth category, attributed to noise and bias in RefCOCO/+/g queries.
Limitations
- Performance depends on the biases and errors of the underlying VLP and OVD models.
- Predicted-category extraction via NLP parsers is noisy on coarse queries such as 'woman red coat' and pure spatial queries, lowering accuracy versus ground-truth category.
- Hyperparameters (GradCAM layer, theta, alpha, m) were tuned on the RefCOCOg validation set.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg Flickr30k Entities COCO class vocabulary
Benchmarks: RefCOCO RefCOCO+ RefCOCOg Flickr30k Entities
Metrics: accuracy Recall@1 Recall@5
Baselines: ReCLIP CPT CPT-adapted NMTree TransVG UNINEXT MDETR GLIP ZSGNet BAN
Builds on: GradCAM ALBEF VinVL Detic ReCLIP
Closest papers by meaning
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.90)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.90)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.90)
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.89)
- SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding (CVPR 2025 · similarity 0.89)
- Revisiting Few-Shot Object Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.