Research map / Vision-language and multimodal / Human and embodied modeling
Look Around Before Locating: Considering Content and Structure Information for Visual Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSRVG, a transformer-based visual grounding model that aligns visual and linguistic content with CLIP-guided knowledge correction and modulates localization via a soft subject/context split of the expression.
Problem
Visual grounding models rely on superficial correlations between expression subjects and common visual entities, and one-stage methods struggle to jointly model visual and linguistic structure due to the gap between continuous image patches and discrete text tokens.
Approach
A cross-modal content alignment encoder whose attention is corrected by a CLIP-derived affinity matrix (knowledge-guided cross-attention), plus a multi-branch decoder that splits the expression into subject and context via dependency parsing and uses them as cascaded queries over visual features.
Key results
- SSRVG-B reaches 90.45% on RefCOCO val and 88.04% on RefCOCOg test-u-level splits as reported in Table 1 (fine-tuned Swin-B), outperforming HiVG on most splits.
- Ablation on RefCOCOg val-u: KGCA adds 1.29 points over the baseline (79.06 to 80.35), and the full model reaches 82.45%.
- Pre-trained SSRVG-B* reports 92.73% on RefCOCO val, exceeding Qwen-VL-7B (89.36%) on that split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SSRVG-B* instead of the strongest listed baseline | visual grounding models · RefCOCO val, testA, and testB | accuracy: 90.77 -> 92.73; 92.94 -> 94.04; 88.03 -> 90.75 | Helped |
| use SSRVG-B* instead of the strongest listed baseline | visual grounding models · RefCOCO+ val, testA, and testB | accuracy: 86.78 -> 85.12; 89.91 -> 88.84; 78.02 -> 78.97 | Mixed |
| use SSRVG-B* instead of the strongest listed baseline | visual grounding models · RefCOCOg val-u and test-u | accuracy: 86.61 -> 88.30; 86.60 -> 88.51 | Helped |
| use SSRVG-B* instead of the strongest listed baseline | visual grounding models · ReferItGame | accuracy: 80.90 -> 77.50 | Hurt |
| use SSRVG-B* instead of the strongest listed baseline | visual grounding models · Flickr30K | accuracy: 82.63 -> 83.15 | Helped |
| add KGCA to the baseline (ablation) | SSRVG-B on RefCOCOg val-u · RefCOCOg val-u | accuracy: 79.06 -> 80.35 | Helped |
| add multi-scale visual features to the baseline (ablation) | SSRVG-B on RefCOCOg val-u · RefCOCOg val-u | accuracy: 79.06 -> 79.80 | Helped |
| add semi-structured reasoning to the baseline (ablation) | SSRVG-B on RefCOCOg val-u · RefCOCOg val-u | accuracy: 79.06 -> 80.11 | Helped |
| combine KGCA and multi-scale features (ablation) | SSRVG-B on RefCOCOg val-u · RefCOCOg val-u | accuracy: 80.61 | Helped |
| combine KGCA and semi-structured reasoning (ablation) | SSRVG-B on RefCOCOg val-u · RefCOCOg val-u | accuracy: 81.04 | Helped |
| remove multi-scale features from the full method (ablation) | SSRVG-B on RefCOCOg val-u · RefCOCOg val-u | accuracy: 81.04 -> 82.45 | Helped |
| remove CLIP guidance (ablation) | SSRVG-B on RefCOCOg val-u · RefCOCOg val-u | accuracy: 80.66 -> 82.45 | Helped |
Surprising findings
- Using CLIP's text encoder instead of BERT drops accuracy by over 11 points, indicating token-level features matter more than CLIP's holistic text representation for grounding.
Limitations
- Subject extraction via Stanza is unreliable on low-quality annotations such as ReferItGame, limiting gains there.
- Lags behind large generalist models (e.g., Qwen-VL, OFA) on RefCOCO+ and RefCOCOg, which use larger parameters and more training data.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg ReferItGame Flickr30K Entities
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReferItGame Flickr30K Entities
Metrics: accuracy (IoU > 0.5)
Baselines: TransVG QRNet VLTVG CLIP-VG TransVG++ VG-LAW LGR-NET EEVG HiVG PolyFormer MDETR OFA
Builds on: DETR TransVG VLTVG UP-DETR Swin Transformer
Closest papers by meaning
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.91)
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.91)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.90)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.90)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.89)
- Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.