Research map / Detection and segmentation / Open-set and few-shot detection
Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PBREC, a point-based two-stage framework for referring expression comprehension that converts boxes and masks into center-ness and mass-ness soft targets, treats grounding and segmentation as point-level binary classification, and uses IoU-based NMS for post-processing.
Problem
Two-stage referring expression comprehension suffers from sparse region proposals that lose spatial context and a train/inference proposal gap, and two-stage segmentation is limited by indirect box-based solutions; one-stage transformer methods are costly to train.
Approach
Replace region-level ranking with a dense point-level search space: expression-aware center-ness and mass-ness soft ground truth turn REC into multi-label binary classification over grid points, with the same architecture serving grounding and segmentation, and an IoU-based NMS selecting the final prediction from top-k proposals.
Key results
- PBREC-MT reaches 82.94/86.31/80.81 top-1 accuracy on RefCOCO val/testA/testB, outperforming the two-stage Ref-NMS by 4.77% on average-style comparisons reported in text.
- On RES, PBREC-MT achieves 71.44/73.21/70.11 overall IoU on RefCOCO, beating two-stage methods by 9.98% to 14.00%.
- Ablation: IoU-based NMS with top-12 gives +1.61% on REG; multi-task soft targets improve grounding by +2.68% over hard labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace hard 0-1 targets with soft center-ness and mass-ness targets (ablation) | PBREC-MT · RefCOCO+ val | accuracy: +1.79% | Helped |
| replace hard 0-1 targets with soft center-ness and mass-ness targets (ablation) | PBREC-MT · RefCOCO+ val | overall IoU: +1.51% | Helped |
| use only center-ness for multi-task training (ablation) | PBREC-MT · RefCOCO+ val | accuracy: +0.58% | Helped |
| use only center-ness for multi-task training (ablation) | PBREC-MT · RefCOCO+ val | overall IoU: -0.21% | Hurt |
| use only mass-ness for multi-task training (ablation) | PBREC-MT · RefCOCO+ val | accuracy: -0.21% | Hurt |
| use only mass-ness for multi-task training (ablation) | PBREC-MT · RefCOCO+ val | overall IoU: +0.20% | Helped |
| apply IoU-based NMS over top-12 proposals instead of top-1 (ablation) | PBREC-MT · RefCOCO+ val | accuracy: +1.61% | Helped |
| expand IoU-based NMS from top-12 to top-16 proposals (ablation) | PBREC-MT · RefCOCO+ val | accuracy: 74.51 -> 74.25 | Hurt |
| use point-based two-stage PBREC-MT instead of Ref-NMS | REG · RefCOCO+ val | accuracy: 68.25 -> 74.85 | Helped |
| use point-based two-stage PBREC-MT instead of Ref-NMS | RES · RefCOCO+ val | overall IoU: 49.76 -> 63.76 | Helped |
Surprising findings
- Conventional two-stage methods with top-1 proposals have a recall ceiling of about 80.77% on RefCOCO testB, which the proposed method exceeds (80.81%).
Limitations
- Relies on pretrained FCOS/SOLO proposal detectors whose recall bounds performance.
- Multi-task soft targets do not consistently help each task (center-ness vs mass-ness trade-offs).
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg
Benchmarks: RefCOCO RefCOCO+ RefCOCOg
Metrics: accuracy at IoU 0.5 overall IoU recall
Baselines: Faster R-CNN Mask R-CNN TransVG RED Word2Pix RefTR Ref-NMS CRIS MAttNet CM-Att
Builds on: FCOS SOLO TransVG Ref-NMS BERT
Closest papers by meaning
- Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension (AAAI 2025 · similarity 0.90)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.89)
- ScanFormer: Referring Expression Comprehension by Iteratively Scanning (CVPR 2024 · similarity 0.89)
- Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.89)
- Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension (NeurIPS 2024 · similarity 0.89)
- 3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.