Research map / Vision-language and multimodal / Human and embodied modeling
Referencing Where to Focus: Improving Visual Grounding with Referential Query
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RefFormer, which adds a query adaptation module inside a frozen CLIP to generate target-related referential queries from multi-level features as priors for a DETR-style grounding decoder.
Problem
DETR-based visual grounding decoders use randomly initialized or linguistic learnable queries that carry no target information and use only the deepest visual features, increasing decoder learning difficulty.
Approach
A query adaptation module inserted at multiple CLIP layers iteratively refines learnable queries with cross-modal and multi-level visual context to produce referential queries that prime the decoder; the module also acts as a residual adapter so CLIP stays frozen.
Key results
- Ours (ViT-Base/16) reaches 86.52 / 90.24 on RefCOCO val / testA and 77.60 on RefCOCOg test, outperforming Dynamic MDETR under the same backbone (e.g., +4.94 on RefCOCOg val).
- Achieves 82.18 on ReferItGame test versus 76.60 for VG-LAW and 82.10 on Flickr30K test.
- Ablation on RefCOCOg shows adding QA at layers {4,6,8,10,12} raises test Prec@0.5 from 65.54 to 75.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add QA at layers {4,6,8,10,12} rather than remove QA (ablation) | RefFormer · RefCOCOg val | Prec@0.5: 65.50 -> 76.33 | Helped |
| use QA at layers {2,4,6,8,10,12} instead of {4,6,8,10,12} (ablation) | RefFormer · RefCOCOg val | Prec@0.5: 76.33 -> 75.84 | Hurt |
| use fusion layers {4,8,12} rather than single-level image features (ablation) | RefFormer decoder · RefCOCOg val | Prec@0.5: 65.42 -> 76.33 | Helped |
| replace multi-level fusion {4,8,12} with fusion layer {4} only (ablation) | RefFormer decoder · RefCOCOg test | Prec@0.5: 75.33 -> 65.12 | Hurt |
| use dual-direction image and text feature flow instead of image-only or text-only flow (ablation) | RefFormer QA module · RefCOCOg val | Prec@0.5: 72.57 -> 76.33 | Helped |
| add auxiliary loss for referential queries (ablation) | RefFormer · RefCOCOg | Prec@0.5 | Helped |
| use QA-generated prior queries instead of random queries or linguistic embeddings (ablation) | RefFormer decoder · RefCOCOg | Prec@0.5 | Helped |
| use Swin-Transformer + BERT single-modal encoders instead of a multi-modal encoder (ablation) | RefFormer · RefCOCOg | Prec@0.5 | No clear effect |
| set the number of learnable queries to 3 (ablation) | RefFormer · RefCOCOg | Prec@0.5 | Helped |
| use RefFormer instead of Dynamic MDETR with the same ViT-Base/16 backbone | RefCOCO · RefCOCO testA | Prec@0.5: +1.42% | Helped |
| use RefFormer instead of Dynamic MDETR | RefFormer · RefCOCO+ testB | Prec@0.5: +6.21% | Helped |
| use RefFormer instead of Dynamic MDETR | RefFormer · RefCOCOg val | Prec@0.5: +4.94% | Helped |
Surprising findings
- The referential query alone often does not precisely localize the target because of the lower channel dimension in the QA module, yet it still captures target-related information.
- Removing the QA module causes a sharp drop on RefCOCOg (about 10 points), indicating the prior queries carry most of the gain.
Limitations
- Performance on referring expression segmentation (RES) is weaker than on REC; the method was not specifically optimized for RES.
- Referential queries are low-dimensional and may not precisely localize the target.
- No error bars reported due to computational cost.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg Flickr30K Entities ReferItGame
Benchmarks: RefCOCO RefCOCO+ RefCOCOg Flickr30K Entities ReferItGame
Metrics: Prec@0.5 mIoU
Baselines: TransVG TransVG++ Dynamic MDETR VG-LAW RefTR QRNeT PVD MDETR
Builds on: DETR CLIP TransVG MaskFormer DAB-DETR
Closest papers by meaning
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.91)
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.91)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.90)
- Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · similarity 0.90)
- Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · similarity 0.90)
- AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.