Research map / Detection and segmentation / Open-set and few-shot detection
Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CCKT-Det, an open-vocabulary detector trained without extra data that injects per-image semantic priors into object queries and distills VLM region features via a regional contrastive loss, forming a cyclic knowledge transfer from CLIP-family teachers.
Problem
Open-vocabulary object detection relies on extra supervision (caption data or pseudo labels) to bridge the misalignment between image-text pretraining and region-level perception; the paper asks whether detectors can align with VLM visual-semantic space without such data.
Approach
Dynamically add CLIP text embeddings of image-present concepts (prefiltered by CLIP or an MLLM) to object queries as language queries, and align matched decoder query embeddings with teacher CLIP region features via a bidirectional regional contrastive loss, forming a cycle between semantic input and visual output.
Key results
- CCKT-Det with SwinB reaches 41.9 AP50 novel on OV-COCO without extra data, close to CORA+ (43.1) which uses COCO captions.
- CCKT-Det++ with a stronger GVT-based prior reaches 46.0 AP50 novel on OV-COCO, the reported new state of the art.
- On OV-LVIS, CCKT-Det++ with RN50 reaches 18.2 APr without extra data; with SwinB and pseudo annotations it reaches 32.8 APr.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic guidance to the detector queries (ablation) | Def-DETR · OV-COCO novel classes | AP50N: 25.4 -> 32.6 | Helped |
| Add regional contrastive distillation loss (ablation) | CCKT-Det with semantic guidance · OV-COCO novel classes | AP50N: 32.6 -> 38.0 | Helped |
| Use similarity-score classification at inference instead of the classification-branch logits (ablation) | CCKT-Det · OV-COCO novel classes | AP50N: 31.7 -> 38.0 | Helped |
| Add semantic guidance to regional contrastive loss and similarity classification (ablation) | CCKT-Det · OV-COCO novel classes | AP50N: 33.7 -> 38.0 | Helped |
| Use a stronger teacher, from ViT-B-16 to ViT-L-14 (ablation) | CCKT-Det with ResNet50 · OV-COCO novel classes | AP50N: 35.2 -> 38.0 | Helped |
| Raise the semantic-prior threshold from 0.70 to 0.90 with 6 priors (ablation) | CCKT-Det · OV-COCO novel classes | AP50N: 38.0 -> 37.6 | Hurt |
| Increase semantic priors from 6 to 12 at threshold 0.70 (ablation) | CCKT-Det · OV-COCO novel classes | AP50N: 38.0 -> 36.8 | Hurt |
| Use 12 semantic priors and threshold 0.90 instead of 6 priors and threshold 0.70 (ablation) | CCKT-Det · OV-COCO novel classes | AP50N: 38.0 -> 36.2 | Hurt |
| Use stronger semantic priors filtered by GVT at inference | CCKT-Det · OV-COCO novel classes | AP50N: 38.0 -> 45.3 | Helped |
| Use CCKT-Det++ rather than BARON for LVIS-trained transfer | LVIS-trained detector · COCO | AP: +2.3% AP | Helped |
| Use CCKT-Det++ rather than BARON for LVIS-trained transfer | LVIS-trained detector · Objects365 | AP: +1.6% AP | Helped |
Surprising findings
- Using the vanilla detection classification head at inference drops performance; the similarity-based CLIP-style classification is needed to realize the gain from the contrastive loss (row 3 vs row 5 of the ablation).
Limitations
- Performance depends on the teacher's classification accuracy; noisy semantic priors (more classes sampled to fill L) degrade results.
- On OV-LVIS the no-extra-data version lags caption-supervised methods, attributed to naive hand-crafted prompts; the strong result needs pseudo-annotations from the model itself.
- Relies on an MLLM or GVT to filter concepts at inference for the best results (CCKT-Det++).
Experimental setup
Datasets: COCO LVIS
Benchmarks: OV-COCO OV-LVIS COCO Objects365
Metrics: AP50 novel (AP50^N) AP rare (APr) AP AP50 AP75
Baselines: Detic OV-DETR RegionCLIP CoDet BARON CORA DITO ViLD F-VLM BIND CLIP-Self OV-DQUO
Builds on: Deformable DETR ViLD OV-DETR CLIP BARON
Closest papers by meaning
- CAKE: Category Aware Knowledge Extraction for Open-Vocabulary Object Detection (AAAI 2025 · similarity 0.91)
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.90)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.90)
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.90)
- Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection (CVPR 2024 · similarity 0.90)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.