Research map / Vision-language and multimodal / CLIP and zero-shot
Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zero-shot referring expression comprehension by parsing captions and images into (subject, predicate, object) triplets with ChatGPT and CLIP/FLAVA, matching them by structural similarity, and fine-tuning the VLA model with a triplet contrastive loss on relationship datasets.
Problem
Zero-shot referring expression comprehension requires disentangling visual scenes and captions into entities and understanding relationships among them, which existing vision-language alignment models like CLIP handle poorly.
Approach
Decompose both image and caption into (subject, predicate, object) triplets, score triplet pairs by summed cosine similarities from a VLA model, propagate triplet-level matches to instance-level similarity, and fine-tune the VLA with LoRA using a triplet-level contrastive loss on HICO-DET, SWiG and Visual Genome.
Key results
- Ours+VR-CLIP improves over ReCLIP by up to 19.53% and 9.74% on average across RefCOCO/+/g splits.
- On Who's Waldo, Ours+VR-CLIP reaches 61.3% test accuracy versus 63.5% for the supervised Who's Waldo method and 29.4% for ReCLIP.
- Removing the subject term from the triplet similarity drops RefCOCOg test accuracy from 59.90 to 47.83.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add triplet-to-instance matching instead of scoring-and-ranking (ablation) | Ours with CLIP · RefCOCOg | accuracy: 55.35 -> 57.60; 54.33 -> 56.64 | Helped |
| fine-tune CLIP as VR-CLIP (ablation) | Ours with triplet matching · RefCOCOg | accuracy: 57.60 -> 59.87; 56.64 -> 59.90 | Helped |
| add the subject similarity term (ablation) | Ours+VR-CLIP · RefCOCOg | accuracy: 48.35 -> 59.87; 47.83 -> 59.90 | Helped |
| add the object similarity term (ablation) | Ours+VR-CLIP · RefCOCOg | accuracy: 56.92 -> 59.87; 57.05 -> 59.90 | Helped |
| add the predicate similarity term and box-pair filtering (ablation) | Ours+VR-CLIP · RefCOCOg | accuracy: 56.90 -> 59.87; 56.81 -> 59.90 | Helped |
| include both terms of the triplet-to-instance aggregation (ablation) | Ours+VR-CLIP · RefCOCOg | accuracy: 59.54 -> 59.87; 59.37 -> 59.90 | Helped |
| include both terms of the triplet-to-instance aggregation (ablation) | Ours+VR-CLIP · RefCOCOg | accuracy: 59.11 -> 59.87; 59.26 -> 59.90 | Helped |
| use structural triplet matching with ChatGPT parsing rather than ReCLIP | CLIP (ViT-B/32) zero-shot REC · RefCOCO/+/g | accuracy | Mixed |
| fine-tune the VLA model on relational triplets as VR-CLIP | Ours with CLIP · RefCOCO/+/g | accuracy | Helped |
| use the proposed zero-shot method rather than ReCLIP | CLIP-based REC · Who’s Waldo Test | accuracy: 29.4 -> 60.8 | Helped |
Surprising findings
- ReCLIP performs no better than random chance on Who's Waldo (29.4% vs. random near chance), indicating its hand-crafted parser fails on long captions.
- Fine-tuning VLA on relationship data reduces CLIP zero-shot image classification accuracy from 0.63 to 0.50.
Limitations
- Fine-tuning sacrifices the generalist zero-shot capabilities of CLIP, e.g., image classification and retrieval drop.
- Relies on ChatGPT for caption and dataset triplet parsing, which is closed-source and costly.
- Evaluated one-to-one grounding only.
Experimental setup
Datasets: HICO-DET SWiG Visual Genome RefCOCO RefCOCO+ RefCOCOg Who's Waldo
Benchmarks: RefCOCO RefCOCO+ RefCOCOg Who's Waldo
Metrics: accuracy (IoU > 0.5)
Baselines: ReCLIP GradCAM CPT-adapted CPT-Blk w/ VinVL CPT-Seg w/ VinVL Gupta et al. SL-CCRF MAttNet UNITER
Builds on: ReCLIP CLIP FLAVA LoRA
Closest papers by meaning
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.89)
- IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis (AAAI 2025 · similarity 0.89)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.88)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.88)
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.88)
- Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.