Research map / Vision-language and multimodal / Human and embodied modeling
Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HieA2G, a grounding network with hierarchical word-object, phrase-object, and text-image alignment plus an adaptive grounding counter, to handle no-target, single-target, and multi-target referring expressions in GREC and related tasks.
Problem
Generalized Referring Expression Comprehension (GREC) requires detecting an arbitrary number of objects (zero to many) from a text expression, which fixed-output REC models and single-level vision-language alignment cannot handle.
Approach
Combine three levels of cross-modal alignment (word-object masked text recovery, phrase-object matching, and global text-image matching) with an adaptive counter that classifies the number of targets into {0,1,2,3,3+} and is trained with a supervised contrastive loss over a memory bank.
Key results
- On gRefCOCO GREC (val), HieA2G-R101 reaches 67.8 Pr and 60.3 N-acc, outperforming UNITEXT (58.2 Pr, 50.6 N-acc).
- On RefCOCO/+/g REC, HieA2G-R101 achieves 87.8/80.7/83.7 on val, exceeding GSVA-7B on most splits.
- Ablation: removing the AGC drops GREC Pr by 13.9% and N-acc by 12.3% relative to the full model per the text; removing L_con lowers Pr from 67.8 to 66.5 and N-acc from 60.3 to 57.3 relative to the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hierarchical word-object, phrase-object, and text-image alignments (ablation) | HieA2G · gRefCOCO | Pr: 65.2 -> 67.8 | Helped |
| Add hierarchical word-object, phrase-object, and text-image alignments (ablation) | HieA2G · gRefCOCO | N-acc.: 54.9 -> 60.3 | Helped |
| Add phrase-object alignment (ablation) | HieA2G · gRefCOCO | Pr: 67.5 -> 67.8 | Helped |
| Add word-object alignment (ablation) | HieA2G · gRefCOCO | Pr: 67.1 -> 67.8 | Helped |
| Add text-image alignment (ablation) | HieA2G · gRefCOCO | Pr: 67.0 -> 67.8 | Helped |
| Use AGC instead of threshold-based output selection (ablation) | HieA2G · gRefCOCO | Pr: 53.9 -> 67.8 | Helped |
| Use AGC instead of threshold-based output selection (ablation) | HieA2G · gRefCOCO | N-acc.: 48.0 -> 60.3 | Helped |
| Add the AGC counting classifier (ablation) | HieA2G · gRefCOCO | Pr: 53.9 -> 66.5 | Helped |
| Add AGC contrastive loss (ablation) | HieA2G · gRefCOCO | Pr: 66.5 -> 67.8 | Helped |
| Add AGC contrastive loss (ablation) | HieA2G · gRefCOCO | N-acc.: 57.3 -> 60.3 | Helped |
Limitations
- Fails on ambiguous visual cues and occluded key objects, producing missed targets or false-negative boxes.
- Counting classifier caps at 3+, so large counts rely on a threshold-based selection.
Experimental setup
Datasets: gRefCOCO RefCOCO RefCOCO+ RefCOCOg Flickr30K Entities
Benchmarks: gRefCOCO RefCOCO RefCOCO+ RefCOCOg Flickr30K Entities
Metrics: Pr@(F1=1, IoU>=0.5) N-acc. T-acc. Recall@k cIoU gIoU oIoU
Baselines: MCN VLT MDETR UNITEXT Ferret MAttNet RefTR SeqTR TransVG++ LISA-7B GSVA-7B LAVT
Builds on: MDETR ReLA bipartite matching
Closest papers by meaning
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.90)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.87)
- Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions (CVPR 2024 · similarity 0.87)
- DViN: Dynamic Visual Routing Network for Weakly Supervised Referring Expression Comprehension (CVPR 2025 · similarity 0.86)
- Revisiting Counterfactual Problems in Referring Expression Comprehension (CVPR 2024 · similarity 0.86)
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.