Research map / Detection and segmentation / Open-set and few-shot detection
CP-DETR: Concept Prompt Guide DETR Toward Stronger Universal Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CP-DETR, a DETR-based universal detector with a prompt-visual hybrid encoder (progressive single-scale fusion plus multi-scale fusion gating) that accepts text, visual-box, and optimized concept prompts within one set of weights.
Problem
Text-prompted universal object detectors suffer alignment bias and underuse prompt information in downstream tasks, leaving them behind specialist models outside pre-training distributions.
Approach
An early-fusion prompt-visual hybrid encoder that fuses prompts with image features scale-by-scale from the deepest level and across all scales, supervised by an auxiliary anchor-based head and a prompt multi-label loss, paired with three concept prompt generators (text via CLIP, visual box-based encoder, and tuned optimized prompts with super-class vectors).
Key results
- CP-DETR-T (Swin-T) reaches 47.6 zero-shot AP on LVIS minival and CP-DETR-L (Swin-L) reaches 32.2 zero-shot AP on ODinW35.
- Visual prompt interactive detection achieves 68.4 AP on COCO val with CP-DETR-L.
- Optimized prompt tuning gives 73.1 average AP on ODinW13, +5.1 over GLIPv2-H.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the prompt visual hybrid encoder instead of a DINO encoder (ablation) | CP-DETR · LVIS minival, zero-shot | AP: 42.2 -> 44.3 | Helped |
| Use the prompt visual hybrid encoder instead of a DINO encoder (ablation) | CP-DETR · ODinW13, full-shot | AP: 58.5 -> 64.0 | Helped |
| Add multi-scale fusion gating (ablation) | CP-DETR · LVIS minival, zero-shot | AP: 42.8 -> 44.3 | Helped |
| Add multi-scale fusion gating (ablation) | CP-DETR · ODinW13, full-shot | AP: 62.3 -> 64.0 | Helped |
| Add prompt multi-label loss (ablation) | CP-DETR · LVIS minival, zero-shot | AP: 44.3 -> 44.8 | Helped |
| Add prompt multi-label loss (ablation) | CP-DETR · ODinW13, full-shot | AP: 64.0 -> 64.2 | Helped |
| Add an auxiliary detection head to the prompt multi-label loss variant (ablation) | CP-DETR · LVIS minival, zero-shot | AP: 44.8 -> 44.7 | Hurt |
| Add an auxiliary detection head to the prompt multi-label loss variant (ablation) | CP-DETR · ODinW13, full-shot | AP: 64.2 -> 64.9 | Helped |
| Add super-class representation to prompt fine-tuning (ablation) | CP-DETR · ODinW13, full-shot | AP: 64.9 -> 67.0 | Helped |
| Increase the optimized prompt super-class representation length (ablation) | CP-DETR-T · Downstream tasks | AP | Helped |
| Use visual prompts instead of text prompts | CP-DETR-L · ODinW35 | AP: +18.4 AP | Helped |
| Use CP-DETR-L with optimized prompts instead of GLIPv2-H with prompt tuning | ODinW13 · ODinW13, prompt fine-tuning | Average AP: +5.1 AP | Helped |
Surprising findings
- Scaling to a larger backbone (EVA-02 ViT-L) in a preliminary model reportedly reaches zero-shot performance rivaling closed-source models, though results are preliminary.
- Pre-training on only publicly available annotations (no pseudo-labeling or private captions) achieves competitive zero-shot performance.
Limitations
- Pre-training depends on text quality; conflicting noun descriptions across datasets hurt optimization.
- Average-pooled sentence-level prompts can mis-optimize sub-phrases (e.g., 'helmet' within 'person wearing helmet').
- Visual prompt performance is notably stronger for the large model than the tiny model, suggesting need for scaling.
Experimental setup
Datasets: Objects365 OpenImages V3Det LVIS COCO GoldG RefCOCO/+/g Visual Genome PhraseCut
Benchmarks: COCO LVIS ODinW35 ODinW13 RefCOCO/+/g
Metrics: AP P@0.5
Baselines: GLIP-T Grounding-DINO-T YOLO-World-L DetCLIPv3-T T-Rex2-T GLEE-Pro APE(D) GLIPv2-H
Builds on: DETR Grounding DINO GLIP CLIP
Closest papers by meaning
- Uni$^2$Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection (ICLR 2025 · similarity 0.88)
- MS-DETR: Efficient DETR Training with Mixed Supervision (CVPR 2024 · similarity 0.87)
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.87)
- DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detection (NeurIPS 2024 · similarity 0.87)
- Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection (NeurIPS 2025 · similarity 0.87)
- DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.