Research map / Vision-language and multimodal / Human and embodied modeling
Investigating Compositional Challenges in Vision-Language Models for Visual Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the ARPGrounding benchmark (attribute, relation, priority) built from Visual Genome to test VLM compositional grounding, shows VLMs near chance on it, and proposes a composition-aware fine-tuning pipeline that uses dependency-parsed text pairs and a heatmap-separation loss.
Problem
Pre-trained vision-language models perform well on weakly supervised visual grounding benchmarks but struggle with compositional reasoning (attributes, relations, and noun priority) when grounding phrases in images.
Approach
Use Grad-CAM-style gradient-weighted heatmaps from VLMs for weakly supervised grounding, build a compositional benchmark from Visual Genome scene graphs, and fine-tune VLMs with a loss that pushes heatmaps for two text descriptions of different objects in the same image to be disjoint, using text pairs chosen via dependency parsing.
Key results
- Pretrained VLMs such as ALBEF and BLIP2 surpass WWbL++ on VG, Flickr, and ReferIt grounding without training (ALBEF mean 76.26 vs 72.28).
- On ARPGrounding, most VLMs score near chance (25%); the best, ALBEF, reaches a mean of 35.13.
- Composition-aware fine-tuning raises ALBEF's ARPGrounding relation score from 29.19 to 38.65 and priority from 14.94 to 24.21.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply composition-aware fine-tuning | CLIP · ARPGrounding Attribute | accuracy: 42.78 to 44.56 | Helped |
| apply composition-aware fine-tuning | CLIP · ARPGrounding Relation | accuracy: 9.19 to 13.24 | Helped |
| apply composition-aware fine-tuning | CLIP · ARPGrounding Priority | accuracy: 11.24 to 21.30 | Helped |
| apply composition-aware fine-tuning | ALBEF · ARPGrounding Attribute | accuracy: 61.25 to 66.34 | Helped |
| apply composition-aware fine-tuning | ALBEF · ARPGrounding Relation | accuracy: 29.19 to 38.65 | Helped |
| apply composition-aware fine-tuning | ALBEF · ARPGrounding Priority | accuracy: 14.94 to 24.21 | Helped |
| fine-tune using text pairs that describe the same object (ablation) | CLIP composition-aware fine-tuning · ARPGrounding Priority | accuracy: 11.24 to 5.63 | Hurt |
| fine-tune using randomly sampled text pairs (ablation) | CLIP composition-aware fine-tuning · ARPGrounding Priority | accuracy: 11.24 to 13.90 | Helped |
| fine-tune using text pairs that describe the same object (ablation) | ALBEF composition-aware fine-tuning · ARPGrounding Attribute | accuracy: 61.25 to 55.71 | Hurt |
| fine-tune using randomly sampled text pairs (ablation) | ALBEF composition-aware fine-tuning · ARPGrounding Relation | accuracy: 29.19 to 35.95 | Helped |
| replace the composition-aware pretext loss with contrastive loss (ablation) | CLIP fine-tuning · ARPGrounding Priority | accuracy: 11.24 to 6.67 | Hurt |
| replace the composition-aware pretext loss with image-text matching loss (ablation) | ALBEF fine-tuning · ARPGrounding Priority | accuracy: 14.94 to 8.82 | Hurt |
Surprising findings
- Fully supervised GLIP and Grounding DINO do no better than weakly supervised VLMs on ARPGrounding.
- Fine-tuning with standard contrastive or image-text matching losses on the generated text pairs reduces grounding performance, while the proposed heatmap loss improves it.
Limitations
- Relation split is small (370 samples) and relies on manual filtering.
- Attribute and priority evaluation uses a single attribute per object and a chance-level metric that depends on sample construction.
- Results limited to four VLMs and fine-tuning on CLIP and ALBEF only.
Experimental setup
Datasets: Visual Genome Flickr30k Entities ReferIt
Benchmarks: VG Flickr30k Entities ReferIt ARPGrounding
Metrics: pointing game accuracy ARPGrounding accuracy
Baselines: WWbL++ CLIP ALBEF METER BLIP2 TSVLC DAC GLIP Grounding DINO
Builds on: Grad-CAM CLIP ALBEF
Closest papers by meaning
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.92)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.91)
- ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs (NeurIPS 2024 · similarity 0.90)
- Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.