Research map / Vision-language and multimodal / Human and embodied modeling
Improved Visual Grounding through Self-Consistent Explanations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finetunes the ALBEF vision-language model with a weakly-supervised self-consistency objective (SelfEQ) on LLM-generated paraphrases so that GradCAM maps for a phrase and its paraphrase agree, improving box-free visual grounding.
Problem
Vision-language models trained only on image-text pairs can localize objects via GradCAM, but their grounding is inconsistent across equivalent phrasings and limited in vocabulary; improving localization without box annotations is the goal.
Approach
A SelfEQ objective that aligns GradCAM attention maps of a caption and its LLM-generated object-centric paraphrase, with an RoI-based consistency loss to avoid trivial solutions, plus a two-level Vicuna-13B prompting pipeline for phrase chunking and paraphrase generation.
Key results
- SelfEQ reaches 84.07% pointing game accuracy on Flickr30k (MS-COCO training), a 4.69-point absolute gain over the best prior weakly-supervised method reported.
- 67.40% on ReferIt (VG training), an absolute improvement of 7.68 points over prior weakly-supervised methods.
- On RefCOCO+ test A/B, SelfEQ reaches 75.10% and 55.49%, vs. 69.35% and 53.77% for off-the-shelf ALBEF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with SelfEQ on Visual Genome image-text pairs | ALBEF · Flickr30k | pointing game accuracy: +4.69% | Helped |
| fine-tune with SelfEQ on Visual Genome image-text pairs | ALBEF · ReferIt | pointing game accuracy: +7.68% | Mixed |
| fine-tune with SelfEQ on MS-COCO image-text pairs | ALBEF · Flickr30k | pointing game accuracy: 84.07 vs 78.10 | Helped |
| fine-tune with SelfEQ on MS-COCO image-text pairs | ALBEF · ReferIt | pointing game accuracy: 62.75 vs 61.53 | Helped |
| fine-tune with SelfEQ | ALBEF · RefCOCO+ Test A | pointing game accuracy: 69.35 -> 75.10 | Helped |
| fine-tune with SelfEQ | ALBEF · RefCOCO+ Test B | pointing game accuracy: 53.77 -> 55.49 | Helped |
| add paraphrases as regular image-text pairs without explicit SelfEQ linkage (ablation) | ALBEF · Flickr30k | pointing game accuracy: 79.38 -> 78.05 | Hurt |
| replace global captions with object-centric phrases (ablation) | ALBEF with Lvl · Flickr30k | pointing game accuracy: 79.90 -> 81.18 | Helped |
| replace global captions with object-centric phrases (ablation) | SelfEQ · Flickr30k | pointing game accuracy: 81.28 -> 84.07 | Helped |
| remove the RoI consistency loss Lcst (ablation) | SelfEQ · Flickr30k | pointing game accuracy: 47.21 -> 81.90 | Helped |
| remove the pixel-wise similarity loss Lsim (ablation) | SelfEQ · Flickr30k | pointing game accuracy: 80.94 -> 81.90 | Helped |
Surprising findings
- Simply adding paraphrases as regular image-text pairs without the SelfEQ objective does not improve performance, and more unstructured paraphrase data even hurts Flickr30k in the data-quantity study.
- SelfEQ comes second to g++ on ReferIt when trained on VG, despite using no box supervision and a simpler architecture.
Limitations
- Paraphrasing strategy only replaces the primary object while keeping other attributes, limiting linguistic diversity.
- Gains over box-supervised methods are not uniform; AMC remains much stronger on RefCOCO+ with box supervision.
- Evaluation relies on pointing game accuracy, which is a coarse localization metric.
Experimental setup
Datasets: Visual Genome MS-COCO Conceptual Captions 3M Flickr30k ReferIt RefCOCO+
Benchmarks: Flickr30k Entities ReferIt RefCOCO+
Metrics: pointing game accuracy
Baselines: ALBEF AMC g++ g GbS MG-ELMo MG-BiLSTM SSS TD VMRM InfoGround Align2Ground
Builds on: ALBEF GradCAM Grounding by Separation AMC
Closest papers by meaning
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.91)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.90)
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.90)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.90)
- Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · similarity 0.89)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.