Research map / Detection and segmentation / Open-set and few-shot detection
How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OVDEval, a 9-sub-task hard-negative benchmark for open-vocabulary detection across six linguistic aspects, and NMS-AP, a class-agnostic NMS fix for the inflated AP problem; evaluates six OVD models.
Problem
Existing OVD evaluations (COCO, LVIS, Flickr30K, RefCOCO) test only object types and referral expressions without hard negatives, and standard AP can be inflated by predicting multiple mutually exclusive labels per box.
Approach
A fine-grained benchmark spanning object, proper noun, attribute, position, relationship, and negation aspects with hard-negative labels, plus NMS-AP, which applies class-ignored NMS to predictions matching ground truth before computing AP so duplicate labels on one object are not rewarded.
Key results
- Grounding DINO achieves 52.50 NMS-AP on COCO-object but only 10.70 NMS-AP on relationship (vs 35.30 traditional AP), showing the inflated AP gap.
- OmDet achieves the best total average NMS-AP of 25.86 across the 9 sub-tasks, while all models score near 0% on celebrity recognition (best 4.60).
- Traditional AP overstates performance by up to ~30% for OVD models on fine-grained sub-tasks, per the authors' analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply class-ignored NMS before AP calculation | OVD models evaluated on OVDEval · Relationship | AP: 35.30 to 10.70 | Hurt |
| Use traditional AP instead of NMS-AP | Grounding DINO · COCO | AP: 52.50 vs. 55.30 | No clear effect |
| Pre-train with attribute data from VAW | OmDet · Color and Material | NMS-AP | Helped |
| Pre-train with RefCOCO orientation data | Grounding DINO · Position | NMS-AP | Helped |
| Pre-train with relation attributes from HOI-A | OmDet · Relationship | NMS-AP | Helped |
| Evaluate fine-grained described detection with conventional AP rather than NMS-AP | OVD models on OVDEval · OVDEval | AP: 30% higher | Hurt |
Surprising findings
- Predictions with highest confidence are frequently incorrect, and models output mutually exclusive labels (e.g., 'cat that is sitting' and 'cat that is not sitting') on the same box.
- Detic scores near zero on celebrity and weak on negation-style long descriptions despite strong COCO performance; MDETR underperforms even on COCO objects.
Limitations
- Evaluation only covers six pre-existing public OVD models with no retraining or new model proposed.
- Proper noun sub-datasets (landmark, logo, celebrity) have limited label sets (9-10 labels).
Experimental setup
Datasets: COCO HICO Visual Genome Laion-400m Objects365 VAW HOI-A RefCOCO
Benchmarks: OVDEval
Metrics: NMS-AP AP AP@[.5:.95]
Baselines: Detic MDETR GLIP FIBER OmDet Grounding DINO
Builds on: COCO AP Grounding DINO GLIP CheckList
Closest papers by meaning
- COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shifts (CVPR 2025 · similarity 0.91)
- OW-OVD: Unified Open World and Open Vocabulary Object Detection (CVPR 2025 · similarity 0.90)
- OVT-B: A New Large-Scale Benchmark for Open-Vocabulary Multi-Object Tracking (NeurIPS 2024 · similarity 0.90)
- The Devil is in the Fine-Grained Details: Evaluating Open-Vocabulary Object Detectors for Fine-Grained Understanding (CVPR 2024 · similarity 0.90)
- Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection (CVPR 2024 · similarity 0.90)
- Taming Self-Training for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.