Research map / Detection and segmentation / Open-set and few-shot detection
Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BIND, a two-stage open-vocabulary detector that learns region-word alignment with a DINOv2-based dual encoder on image-text pairs, then trains a DETR-style built-in decoder with an adaptive anchor proposal network, avoiding transfer to off-the-shelf detectors.
Problem
Open-vocabulary object detection relies on transferring CLIP-style vision-language knowledge to external detectors such as Faster-RCNN, which loses information due to domain gap and uses non-adaptive manual anchors that are redundant and slow.
Approach
Learn fine-grained region-word alignment directly in a built-in encoder-decoder detector (dual encoder with Hungarian bipartite matching and focal loss), and replace manual anchors with an anchor proposal network that adaptively selects positional anchors from image patches and query embeddings.
Key results
- BIND (ViT-B/16) reaches 36.3 novel AP50 on OV-COCO, and the ViT-L/16 variant reaches 41.5 novel AP50, reported as outperforming prior methods.
- On OV-LVIS, BIND (ViT-B/16) achieves 29.4 APr and the ViT-L/16 variant achieves 32.5 APr.
- Replacing the anchor proposal network with DETR-style 100 prediction slots drops novel AP50 from 36.3 to 32.4 and raises inference time from 0.21 to 0.63 s/img.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BIND with ViT-B/16 instead of CORA | OV-COCO novel detection · OV-COCO novel | AP50: 35.1 -> 36.3 | Helped |
| Scale BIND from ViT-B/16 to ViT-L/16 | BIND · OV-COCO novel | AP50: 36.3 -> 41.5 | Helped |
| Use BIND with ViT-B/16 instead of CFM-ViT | OV-LVIS rare detection · OV-LVIS rare | APr: 28.8 -> 29.4 | Helped |
| Scale BIND from ViT-B/16 to ViT-L/16 | BIND · OV-LVIS rare | APr: 29.4 -> 32.5 | Helped |
| Use the ViT-L/16 BIND model instead of CORA RN50×4 | OV-COCO novel detection · OV-COCO novel | AP50: 41.7 -> 41.5 | Hurt |
| Use BIND with ViT-B/16 instead of CORA RN50×4 | OV-COCO inference · OV-COCO | seconds per image: 0.50 -> 0.21 | Helped |
| Use BIND with ViT-L/16 instead of CORA RN50×4 | OV-COCO inference · OV-COCO | seconds per image: 0.50 -> 0.33 | Helped |
| Add the DINOv2 ViT-B/16 visual encoder instead of CLIP (ablation) | BIND · OV-COCO novel | AP50: 34.2 -> 36.3 | Helped |
| Replace the anchor proposal network with prediction slots (ablation) | BIND · OV-COCO novel | AP50: 36.3 -> 32.4 | Hurt |
| Replace the anchor proposal network with prediction slots (ablation) | BIND · OV-COCO inference | seconds per image: 0.21 -> 0.63 | Hurt |
| Add region-word alignment to CLIP (R50) (ablation) | BIND · OV-COCO novel | AP50: 24.5 -> 33.8 | Helped |
| Add region-word alignment to CLIP (ViT-B/16) (ablation) | BIND · OV-COCO novel | AP50: 25.1 -> 34.2 | Helped |
Surprising findings
- CLIP with ViT-B/16 gets only 25.1 novel AP50 without region-word alignment, indicating region-word alignment is critical for detection.
- CLIP ImageNet whole-image accuracy is 60% top-1 but only 19% on LVIS region classification, showing a region-level gap.
Limitations
- Relies on an off-the-shelf RPN during region-word alignment training, trained only on base class data.
- Reported gains rely on large pretraining data and bigger backbones; scalability shown only up to 3.3M pairs.
Experimental setup
Datasets: CC3M COCO LVIS
Benchmarks: OV-COCO OV-LVIS
Metrics: AP50 novel AP50 mAP APr seconds per image
Baselines: OVR-CNN ViLD Detic OV-DETR RegionCLIP BARON CORA OADP OWL-ViT MEDet Ro-ViT CFM-ViT
Builds on: DETR OV-DETR CLIP DINOv2 Faster R-CNN
Closest papers by meaning
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.91)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.90)
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.90)
- DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.90)
- DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection (CVPR 2024 · similarity 0.89)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.