Research map / Detection and segmentation / Open-set and few-shot detection
RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RefDetector, a matching-based referring expression comprehension method that uses top-k proposal selection, a transformer-style visual-semantic interaction module, and three alignment losses on frozen detectors and a frozen CLIP text encoder.
Problem
Matching-based referring expression comprehension suffers from proposal mismatch caused by confidence thresholding and from complicated, structure-specific fusion modules that depend on parsing expressions into predefined roles.
Approach
Replace fixed confidence thresholds with a top-k proposal strategy to raise referent recall, use a simple stack of attention-based visual and semantic interaction modules instead of manually designed fusion, and apply DETR learnable queries as proposal features while keeping detector and text encoder frozen.
Key results
- RefDetector with DETR (RN101) reaches 87.21 on RefCOCO val and 76.43 on RefCOCOg test (Prec@0.5).
- RefDetector with DETR (RN50) surpasses VG-LAW by up to +2.24 points on RefCOCOg val, despite freezing the detector and linguistic backbone.
- Ablation on RefCOCOg shows that adding the visual interaction, semantic interaction, and referent matching losses raises DETR-based test accuracy from 71.77 to 76.43 when combined with the semantic component.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual interaction (ablation) | RefDetector on RefCOCOg · RefCOCOg | Prec@0.5: 72.77 -> 77.00 | Helped |
| add semantic interaction (ablation) | RefDetector on RefCOCOg · RefCOCOg | Prec@0.5: 72.00 -> 77.00 | Helped |
| add position encoding to visual interaction (ablation) | RefDetector with FasterRCNN on RefCOCOg · RefCOCOg | Prec@0.5: 69.37 -> 70.98 | Helped |
| use sinusoidal rather than location position encoding (ablation) | RefDetector with DETR on RefCOCOg · RefCOCOg | Prec@0.5: 75.61 -> 76.43 | Helped |
| add EPM to EPCL and HPL (ablation) | RefDetector with DETR on RefCOCOg · RefCOCOg | Prec@0.5: 76.11 -> 77.00 | Helped |
| add HPL to EPCL and EPM (ablation) | RefDetector with DETR on RefCOCOg · RefCOCOg | Prec@0.5: 75.06 -> 77.00 | Helped |
| add EPM to EPCL and HPL (ablation) | RefDetector with FasterRCNN on RefCOCOg · RefCOCOg | Prec@0.5: 70.98 -> 71.39 | Helped |
| add HPL to EPCL and EPM (ablation) | RefDetector with FasterRCNN on RefCOCOg · RefCOCOg | Prec@0.5: 69.41 -> 71.39 | Helped |
| combine C2 RoI features with learnable queries (ablation) | RefDetector with DETR on RefCOCOg · RefCOCOg | Prec@0.5: 75.86 -> 77.00 | Helped |
| use C2 rather than C4 RoI features with learnable queries (ablation) | RefDetector with DETR on RefCOCOg · RefCOCOg | Prec@0.5: 74.85 -> 77.00 | Helped |
| replace MHSA with cross-attention to the image feature map (MHCA) (ablation) | RefDetector with DETR on RefCOCOg · RefCOCOg | Prec@0.5: 75.32 -> 77.00 | Helped |
| replace MHSA with cross-attention to the image feature map (MHCA) (ablation) | RefDetector with FasterRCNN on RefCOCOg · RefCOCOg | Prec@0.5: 69.75 -> 71.39 | Helped |
Surprising findings
- Low-level ResNet C2 RoI features outperform the commonly used C4 features for proposal representation in this setup.
Limitations
- Applied only to the REC task; extension to open-vocabulary detection and referring segmentation is future work.
- Interaction module design is intentionally simple and not extensively optimized.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg
Benchmarks: RefCOCO RefCOCO+ RefCOCOg
Metrics: Prec@0.5
Baselines: VG-LAW Ref-NMS MAttNet CM-A-E RvG-Tree TransVG RefTR QRNeT SeqTR
Builds on: DETR Faster R-CNN Mask R-CNN Ref-NMS CLIP
Closest papers by meaning
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.92)
- ScanFormer: Referring Expression Comprehension by Iteratively Scanning (CVPR 2024 · similarity 0.91)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.90)
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.89)
- Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.89)
- A Large-Scale Human-Centric Benchmark for Referring Expression Comprehension in the LMM Era (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.