Research map / Detection and segmentation / Weakly supervised detection and segmentation
Weakly Supervised Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WSOVOD, a weakly supervised open-vocabulary detector trained on multiple image-level-labeled datasets, combining dataset-aware feature adaptation, SAM-guided proposals, and text-embedding-based multiple-instance learning.
Problem
Weakly supervised object detection is limited to closed-set categories from a single dataset; the paper extends WSOD to open-vocabulary detection using only image-level labels across diverse datasets.
Approach
Uses an input-conditional data-aware feature extractor that combines dataset attribute prototypes to correct dataset bias, a SAM-augmented location-oriented WSRPN for proposals, and a MIL network whose classification stream is initialized with frozen text embeddings of category names, with SAM refining boxes between refinement branches.
Key results
- WSOVOD with RN50-WS-MRRP reaches 63.4% mAP on VOC 2007 and 62.1% on VOC 2012 under image-level supervision, a new WSOD state of the art.
- Training WSOVOD on COCO with image-level labels yields 35.0 AP_N on the 17 novel COCO classes, outperforming OVR-CNN (22.8) and ViLD (27.6) which use box-level base annotations.
- Adding DAFE improves CorLoc on VOC07 by 3.0 points when training on VOC07+VOC12.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train jointly on COCO and LVIS rather than COCO alone | WSOVOD · MS COCO novel classes | AP_N: 35.0 -> 36.7 | Helped |
| add VOC07 and VOC12 training data to COCO | WSOVOD · MS COCO novel classes | AP_N: 15.4 -> 19.2 | Helped |
| add ILSVRC training data to COCO | WSOVOD · MS COCO novel classes | AP_N: 35.0 -> 35.6 | Helped |
| add COCO training data to LVIS | WSOVOD · PASCAL VOC 2007 | mAP: 31.0 -> 60.5 | Helped |
| add COCO training data to LVIS | WSOVOD · PASCAL VOC 2007 | CorLoc: 44.5 -> 78.2 | Helped |
| add BCAS sampling for LVIS | WSOVOD · MS COCO | AP 0.5: 12.9 -> 16.7 | Helped |
| add DAFE (ablation) | WSOVOD trained on VOC07 and VOC12 · PASCAL VOC 2007 | mAP: 63.5 -> 64.1 | Helped |
| add DAFE (ablation) | WSOVOD trained on VOC07 and VOC12 · PASCAL VOC 2007 | CorLoc: 79.2 -> 82.2 | Helped |
| add DAFE (ablation) | WSOVOD trained on VOC07 and COCO · PASCAL VOC 2007 | mAP: 61.4 -> 63.0 | Helped |
| add DAFE (ablation) | WSOVOD trained on VOC07 and COCO · PASCAL VOC 2007 | CorLoc: 78.2 -> 80.5 | Helped |
| add SAM proposals and refinement to LO-WSRPN (ablation) | WSOVOD · PASCAL VOC 2007 | mAP: 46.7 -> 63.0 | Helped |
| add SAM proposals and refinement to LO-WSRPN (ablation) | WSOVOD · PASCAL VOC 2007 | CorLoc: 65.1 -> 81.0 | Helped |
Surprising findings
- Removing COCO captions from Detic drops its novel-class AP_N from 27.8 to 1.3, showing fully-supervised open-vocabulary methods depend heavily on caption supervision.
- Combining LVIS with COCO yields only modest gains over COCO alone on VOC07 (61.7 vs 60.5 mAP), while LVIS alone is weaker than COCO alone.
Limitations
- Performance on novel categories in LVIS and ILSVRC remains low (e.g., 13.2 AP and 7.0 AP on COCO when trained on those datasets alone).
- LVIS training saturates early due to incomplete federated image-level annotations, requiring batch-class-aware sampling.
Experimental setup
Datasets: Pascal VOC 2007 Pascal VOC 2012 MS COCO ILSVRC LVIS
Benchmarks: Pascal VOC 2007 Pascal VOC 2012 MS COCO
Metrics: mAP CorLoc AP AP_N AP_B AP_0.5 AP_0.75 recall
Baselines: WSDDN OICR UWSOD MIST Seo et al. OVR-CNN ViLD Detic SGDN PBBL
Builds on: WSDDN OICR UWSOD Segment Anything Faster R-CNN
Closest papers by meaning
- Weakly Supervised Few-Shot Object Detection with DETR (AAAI 2024 · similarity 0.88)
- Open-Vocabulary Object Detection via Language Hierarchy (NeurIPS 2024 · similarity 0.88)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.88)
- OW-OVD: Unified Open World and Open Vocabulary Object Detection (CVPR 2025 · similarity 0.87)
- SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection (CVPR 2024 · similarity 0.87)
- ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.