Research map / Detection and segmentation / Open-set and few-shot detection
Simple Image-Level Classification Improves Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a CLIP-distilled image-level multi-label recognition module whose scores are geometrically fused with instance scores of existing open-vocabulary detectors to improve detection of hard novel and base objects.
Problem
Open-vocabulary object detectors rely on regional features and under-use CLIP's global scene understanding, which hurts detection of small, blurred, or occluded objects that depend on context, especially novel ones.
Approach
Train a plug-in multi-label recognition module on image-level labels with a text-branch (rank loss on CLIP text embeddings) and a visual-branch (L1 distillation from the CLIP image encoder), fuse their sigmoid scores, and refine detector instance scores with the image-level MLR scores via weighted geometric mean.
Key results
- Plugging SIC-CADS into POMP reaches 26.6 AP_r on OV-LVIS (+1.4), and gains of 1.4-3.9 AP_r across five OVOD models on OV-LVIS.
- On OV-COCO, SIC-CADS improves AP_novel by 1.7-3.2 across five OVOD models.
- Cross-dataset: +2.1 AP50 on Objects365 and +3.9 on OpenImages for BoxSup; ViT-L/14 CLIP variant raises AP_r to 21.9 on BoxSup (+5.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SIC-CADS to POMP | POMP · OV-LVIS | AP_r: 25.2 -> 26.6; +1.4 | Helped |
| Add SIC-CADS to CORA | CORA · OV-COCO | AP_novel: 41.6 -> 43.3; +1.7 | Helped |
| Add SIC-CADS to Detic | Detic · Objects365 | AP_50: 29.3 -> 31.2; +1.9 | Helped |
| Add SIC-CADS to Detic | Detic · OpenImages | AP_50: 53.2 -> 54.7; +1.5 | Helped |
| Add Text MLR (ablation) | BoxSup · OV-LVIS | AP_r: 16.4 -> 17.6; +1.2 | Helped |
| Add Visual MLR (ablation) | BoxSup · OV-LVIS | AP_r: 16.4 -> 20.1; +3.7 | Helped |
| Add Visual MLR (ablation) | BoxSup · OV-LVIS | AP_f: 35.4 -> 34.7; -0.7 | Hurt |
| Combine Text MLR and Visual MLR as Multi-modal MLR (ablation) | BoxSup · OV-LVIS | AP_r: 16.4 -> 19.9; +3.5 | Helped |
| Use Visual MLR+ in place of Visual MLR (ablation) | BoxSup · OV-LVIS | AP_r: 20.1 -> 20.6 | Helped |
| Replace ViT-B/32 CLIP with ViT-L/14 CLIP (ablation) | Multi-modal MLR+ with BoxSup · OV-LVIS | AP_r: +5.5 | Helped |
| Remove score normalization (ablation) | SIC-CADS with BoxSup · OV-LVIS | AP_r: 20.3 -> 18.0 | Hurt |
| Use online rather than offline training (ablation) | SIC-CADS with BoxSup · OV-LVIS | AP_r: 19.9 -> 19.2 | Hurt |
Surprising findings
- Using the CLIP image encoder directly at test time (visual MLR+) works better than the distilled student, showing a gap between student and teacher.
- Online (joint) training of the MLR module with the detector underperforms offline training by about 1.3 AP_r on OV-LVIS, attributed to multi-task conflicts.
Limitations
- Fails when context information does not match the object category, e.g., keyboard or pencil in unusual scenes.
- Online joint training reduces performance due to multi-task conflicts.
- Inference cost increases modestly, and gains on frequent categories are near zero.
Experimental setup
Datasets: LVIS v1.0 COCO ImageNet-21k COCO Captions CC3M Objects365 OpenImages
Benchmarks: OV-LVIS OV-COCO Objects365 OpenImages
Metrics: AP_r AP_c AP_f AP AP_novel AP_base AP50 recall of top-10 MLR predictions GFLOPs FPS
Baselines: BoxSup ViLD RegionCLIP OC-OVD Detic POMP BARON CORA
Builds on: CLIP ViLD Detic BoxSup RegionCLIP
Closest papers by meaning
- CAKE: Category Aware Knowledge Extraction for Open-Vocabulary Object Detection (AAAI 2025 · similarity 0.90)
- ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.89)
- SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection (CVPR 2024 · similarity 0.89)
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.89)
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.89)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.