Research map / Detection and segmentation / Open-set and few-shot detection
DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DetCLIPv3, an open-vocabulary detector augmented with an object captioner that generates hierarchical object labels, trained on a 50M-pair auto-annotated dataset (GranuCap50M) with a low-res pretraining then high-res finetuning schedule.
Problem
Open-vocabulary object detectors require a predefined category list at inference and cannot produce descriptive hierarchical labels for objects without category names.
Approach
Grounds a DETR-style open-vocabulary detector with a Q-Former-like object captioner that generates 'phrase | category | parent category' labels per detected object, with training data produced by a VLLM-based recaptioning and entity extraction pipeline and pseudo-boxes from the detector.
Key results
- Swin-T DetCLIPv3 achieves 47.0 zero-shot fixed AP on LVIS minival, outperforming DetCLIPv2 by 6.6 AP.
- Swin-L DetCLIPv3 reaches 19.7 AP dense captioning on VG V1.2, surpassing GRiT (15.5) by 4.2 AP.
- Zero-shot generative detection on COCO novel classes reaches 54.7 AP50 (Swin-T) versus 43.3 for DetPro, though overall AP50 is lower than discriminative baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DetCLIPv2 with DetCLIPv3 | Swin-T open-vocabulary detector · LVIS minival | fixed AP: 40.4 -> 47.0 | Helped |
| Replace DetCLIPv2 with DetCLIPv3 | Swin-L open-vocabulary detector · LVIS minival | fixed AP: 44.7 -> 48.8 | Helped |
| Use zero-shot generative detection | DetCLIPv3 · COCO novel classes | AP50: 43.3 -> 54.7 | Helped |
| Use DetCLIPv3 for dense captioning | DetCLIPv3 · VG V1.2 | AP: 15.5 -> 18.4 | Helped |
| Add GoldG training data (ablation) | Objects365-only Swin-T detector · LVISmini | APall: 30.8 -> 41.4 | Helped |
| Add V3Det training data (ablation) | Detector trained on Objects365 and GoldG · LVISmini | APall: 41.4 -> 42.5 | Helped |
| Add 600k GranuCap samples (ablation) | Detector trained on Objects365, GoldG, and V3Det · LVISmini | APall: 42.5 -> 45.3 | Helped |
| Add 600k GranuCap samples (ablation) | Detector trained on Objects365, GoldG, and V3Det · COCO-O | AP: 30.7 -> 36.4 | Helped |
| Add the object captioner (ablation) | DetCLIPv3 detector · LVISmini | APall: 45.3 -> 46.6 | Helped |
| Add stage 2 captioner pretraining (ablation) | DetCLIPv3 · VG V1.2 | AP: 17.1 -> 18.4 | Helped |
| Add stage 2 captioner pretraining (ablation) | DetCLIPv3 · LVISmini | APall: 46.6 -> 47.0 | Helped |
| Set pseudo-label filtering threshold to 0.2 rather than 0.15, 0.25, or 0.3 (ablation) | Swin-T detector with pseudo-labeled image-text pairs · LVISmini | APall: 45.0; 45.3; 44.8; 44.9 | Helped |
Surprising findings
- Adding the object captioner without any new data raised LVIS AP from 45.3 to 46.6, suggesting generative training improves detector features.
- Swin-L model's rare-category AP on LVIS minival (49.9) exceeds its frequent and common AP.
Limitations
- Evaluation of generative detection capability is incomplete because existing benchmarks do not effectively assess generative detection results.
- Detection process cannot be controlled by instructions.
- Zero-shot generative COCO evaluation relies on post-processing such as category matching and similarity thresholds.
Experimental setup
Datasets: Objects365 V3Det GoldG GranuCap50M CC15M YFCC LAION Visual Genome COCO LVIS ODinW COCO-O
Benchmarks: LVIS minival LVIS val COCO COCO-O COCO (OV setting) VG V1.2 VG-COCO ODinW13
Metrics: fixed AP AP50 dense captioning AP effective robustness
Baselines: GLIP GLIPv2 GroundingDINO DetCLIPv2 OWL-ST CapDet GRiT DetPro ViLD Detic
Builds on: DetCLIPv2 GroundingDINO Q-Former DetCLIP InstructBLIP
Closest papers by meaning
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.91)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.91)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.90)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.89)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.89)
- Hyperbolic Learning with Synthetic Captions for Open-World Detection (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.