Research map / Detection and segmentation / Open-set and few-shot detection
Retrieval-Augmented Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RALF, which adds retrieval-augmented triplet losses using hard/easy negative vocabularies and LLM-generated verbalized concepts that augment visual features, plugged into existing open-vocabulary detectors to improve novel-class detection on COCO and LVIS.
Problem
Open-vocabulary object detection relies on positive pseudo-labels and class names, leaving negative vocabularies and descriptive visual attributes underused for generalizing to novel categories.
Approach
Retrieves semantically similar (hard) and dissimilar (easy) negative class names from a large vocabulary store to define triplet losses (RAL), and augments visual region features with retrieved LLM-generated verbalized concepts through a small decoder (RAF), used as a plug-in ensemble at inference.
Key results
- Object-Centric-OVD + RALF reaches 41.3 APN50 on COCO novel categories, up from 36.6 for the baseline.
- OADP + RALF improves COCO novel APN50 from 30.0 to 33.4.
- OADP + RALF reaches 21.9 APr on LVIS, up from 19.9, with mask AP all improved to 26.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RALF to Object-Centric-OVD | Object-Centric-OVD · COCO | APN50: 36.6 -> 41.3 | Helped |
| Add RALF to OADP | OADP · COCO | APN50: 30.0 -> 33.4 | Helped |
| Add RALF to OADP | OADP · LVIS | APr: 19.9 -> 21.9 | Helped |
| Add RALF to DetPro | DetPro · LVIS | APr: 19.8 -> 21.1 | Helped |
| Add RALF to Object-Centric-OVD | Object-Centric-OVD · LVIS | APr: 17.1 -> 18.5 | Helped |
| Add RAL to OADP without RAF (ablation) | OADP · COCO | APN50: 30.0 -> 31.5 | Helped |
| Add RAF to OADP without RAL (ablation) | OADP · COCO | APN50: 30.0 -> 31.3 | Helped |
| Add RAL to Object-Centric-OVD without RAF (ablation) | Object-Centric-OVD · COCO | APN50: 40.0 -> 40.8 | Helped |
| Add RAF to Object-Centric-OVD without RAL (ablation) | Object-Centric-OVD · COCO | APN50: 40.0 -> 41.0 | Helped |
| Add RAL to Object-Centric-OVD without RAF (ablation) | Object-Centric-OVD · LVIS | APr: 18.1 -> 19.5 | Helped |
| Add RAF to Object-Centric-OVD without RAL (ablation) | Object-Centric-OVD · LVIS | APr: 18.1 -> 18.5 | Helped |
| Add RAL to DetPro without RAF (ablation) | DetPro · LVIS | APr: 19.9 -> 20.5 | Helped |
Surprising findings
- Object-Centric-OVD + RALF slightly decreases LVIS APc, APf and AP while improving APr.
Limitations
- Gains on novel categories are modest for some baselines and LVIS all-category AP sometimes drops.
- Relies on a large external vocabulary store (V3Det) and LLM-generated concepts, adding dependency on external resources.
Experimental setup
Datasets: COCO LVIS V3Det
Benchmarks: COCO LVIS
Metrics: AP50 novel (APN50) AP50 base (APB50) AP50 all mask AP rare (APr) APc APf AP
Baselines: ViLD PB-OVD OV-DETR VL-Det MEDet BARON OADP Object-Centric-OVD DetPro
Builds on: OADP Object-Centric-OVD DetPro CLIP ViLD
Closest papers by meaning
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.88)
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.87)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.87)
- OW-OVD: Unified Open World and Open Vocabulary Object Detection (CVPR 2025 · similarity 0.86)
- OVMR: Open-Vocabulary Recognition with Multi-Modal References (CVPR 2024 · similarity 0.86)
- The Devil is in the Fine-Grained Details: Evaluating Open-Vocabulary Object Detectors for Fine-Grained Understanding (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.