Research map / Detection and segmentation / Open-set and few-shot detection

The Devil is in the Fine-Grained Details: Evaluating Open-Vocabulary Object Detectors for Fine-Grained Understanding

CVPR 2024 · Detection and segmentation › Open-set and few-shot detection · approach: Open-vocabulary object detection · 16 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an evaluation protocol and PACO-derived benchmark suite that probes open-vocabulary detectors' ability to discriminate fine-grained object attributes (color, material, pattern, transparency) using LLM-written positive captions and attribute-substituted hard negatives.

Problem

Open-vocabulary object detectors are evaluated on generic class names (COCO, LVIS), so it is unknown whether they can discern fine-grained extrinsic object properties like color, material, and pattern when given detailed text labels with hard negatives.

Approach

Per-object dynamic vocabularies consisting of one LLM-generated positive caption and several attribute-substituted negative captions, evaluated with class-agnostic NMS and COCO mAP plus a median-rank metric, organized into difficulty-based and attribute-based benchmarks.

Key results

• On the Hard difficulty split with N=5 negatives, OWL (B/16) and OWL (L/14) reach about 26.2 and 26.5 mAP, while Detic drops to 11.5 despite 69.7 mAP on Trivial.
• OWLv2 (L/14) achieves the best color mAP (53.3), but CORA reaches only 25.0 color and 19.3 material mAP on attribute-based benchmarks.
• Performance on standard LVIS rare categories does not correlate with fine-grained hard-negative performance; OWLv2 gains almost nothing over OWL on the proposed benchmarks.

Measured effects reported in the paper

Surprising findings

• Detic is the best detector on Trivial negatives but the worst on Hard negatives, despite strong LVIS performance.
• OWLv2, pretrained on 10B web images, shows almost no gain over OWL on fine-grained discrimination.

Limitations

• OWL and OWLv2 were evaluated only on captions within their 16-token limit (about 80% of captions).
• GroundingDINO required one forward pass per caption because it cannot take a vocabulary directly.
• Positive captions were manually spot-checked rather than fully verified.

Experimental setup

Datasets: PACO COCO LVIS

Benchmarks: FG-OVD Hard FG-OVD Medium FG-OVD Easy FG-OVD Trivial FG-OVD Color FG-OVD Material FG-OVD Pattern FG-OVD Transparency LVIS rare

Metrics: mAP median rank

Baselines: OWL OWLv2 Detic ViLD GroundingDINO CORA

Builds on: PACO OWL-ViT GroundingDINO Detic ViLD

Closest papers by meaning

• GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.92)
• How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.90)
• Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.89)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
• Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.89)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)