Research map / Detection and segmentation / Weakly supervised detection and segmentation
Open-Vocabulary Object Detection via Language Hierarchy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DetLH, which expands ImageNet-21K image-level labels with WordNet hierarchy to co-regularize self-training for weakly-supervised open-vocabulary detectors, and uses WordNet-matched CLIP prompts at test time, evaluated zero-shot across 14 detection datasets.
Problem
Weakly-supervised object detection with image-level labels suffers from image-to-box label mismatch, since image-level labels do not convey precise object information; there is also a vocabulary gap between training and test concepts.
Approach
Expand image-level labels using WordNet hypernyms and hyponyms to provide richer supervision, and co-regularize this with box-level self-training, weighting expanded labels by predicted reliability; additionally map test class names to WordNet synsets via CLIP embedding similarity to generate prompts.
Key results
- On Object365 zero-shot, DetLH reaches 23.6 AP vs 21.6 for Detic (Swin-B).
- Averaged over 14 datasets, DetLH achieves 34.6 AP and 49.3 AP50 vs 31.0 AP and 44.0 AP50 for Detic.
- Ablation on Object365 (AP50): LHST alone 31.3, LHPG alone 31.0, both 32.5, versus box-supervised baseline 26.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LHST and LHPG to the detector (ablation) | DetLH · Object365 | AP50: 26.5 -> 32.5 | Helped |
| Add LHST (ablation) | DetLH · Object365 | AP50: 26.5 -> 31.3 | Helped |
| Add LHPG (ablation) | DetLH · Object365 | AP50: 26.5 -> 31.0 | Helped |
| Vary the pseudo-box confidence threshold from 0.65 to 0.85 (ablation) | LHST · Object365 | AP50: 31.1; 31.3; 31.3; 31.3; 31.2 | No clear effect |
| Use DetLH instead of Detic | Swin-B detector · 14 detection datasets | AP: 31.0 -> 34.6 | Helped |
| Use DetLH instead of Detic | Swin-B detector · Object365 | AP50: 29.4 -> 32.5 | Helped |
| Use DetLH instead of DLWL | Swin-B detector · BDD100K-weather foggy | AP50: 40.2 -> 37.1 | Hurt |
| Use DetLH instead of Detic | Swin-B detector · Pascal VOC | AP: 62.4 -> 64.4 | Helped |
Surprising findings
- DetLH surpasses the dataset-specific oracle on Pascal VOC (64.4 AP vs 54.4 AP).
Limitations
- Expanded labels are restricted to the closed ImageNet-21K vocabulary.
- Reliability of WordNet-expanded labels is not guaranteed; some expanded labels are noisy.
- Dataset details and implementation specifics are deferred to an appendix not included in the text.
Experimental setup
Datasets: LVIS ImageNet-21K Object365 Pascal VOC Cityscapes Vistas SODA10M BDD100K DAWN MIO-TCD BAAI-VANJEE DETRAC
Benchmarks: Object365 Pascal VOC Cityscapes Vistas SODA10M BDD100K DAWN MIO-TCD BAAI-VANJEE DETRAC UAVDT Arthropod Detection
Metrics: AP AP50 AP75 APs APm APl
Baselines: WSDDN YOLO9000 DLWL Detic
Builds on: Detic CenterNet2 CLIP
Closest papers by meaning
- Taming Self-Training for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.89)
- Language-conditioned Detection Transformer (CVPR 2024 · similarity 0.89)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.88)
- SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection (CVPR 2024 · similarity 0.88)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.88)
- Weakly Supervised Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.