Research map / Detection and segmentation / Open-set and few-shot detection
LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DVDet, which adds a conditional context visual prompt to open-vocabulary detectors and iteratively mines LLM-generated fine-grained part descriptors to improve region-text alignment.
Problem
Open-vocabulary object detectors align region embeddings only with categorical labels, missing the fine-grained part/attribute descriptor alignment that VLMs are good at, limiting transfer of VLM knowledge to detection.
Approach
A conditional context prompt (CCP) that expands each RoI with surrounding context and uses a meta-network to produce a per-proposal prompt, making RoI features image-like; combined with an interactive, hierarchical LLM loop that keeps high-frequency descriptors and queries LLMs about confusing categories, with cosine-similarity merging of duplicate descriptors.
Key results
- On OV-COCO, DVDet raises VLDet novel AP from 32.0 to 34.6 and RegionCLIP novel AP from 26.8 to 28.4.
- On OV-LVIS with RN50, DVDet improves VLDet mAP_novel from 21.7 to 23.1 and ViLD from 16.6 to 18.7.
- Transfer from COCO to PASCAL VOC improves VLDet from 61.7 to 64.0 mAP without retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DVDet to BARON | BARON · OV-COCO novel categories | AP: 33.1 -> 35.8 | Helped |
| Add DVDet to VLDet | VLDet · OV-LVIS novel categories (RN50) | mask AP: 21.7 -> 23.1 | Helped |
| Add DVDet to VLDet | VLDet · OV-LVIS novel categories (Swin-B) | mask AP: 26.3 -> 27.5 | Helped |
| Use Templates H and C descriptors without conditional context prompt (ablation) | VLDet and RegionCLIP · OV-COCO novel categories | AP: 32.0 -> 29.7; 26.8 -> 24.6 | Hurt |
| Add conditional context prompt without descriptor templates (ablation) | VLDet and RegionCLIP · OV-COCO novel categories | AP: 32.0 -> 33.1; 26.8 -> 27.3 | Helped |
| Use Template C descriptors with conditional context prompt (ablation) | VLDet and RegionCLIP · OV-COCO novel categories | AP: 32.0 -> 34.0; 26.8 -> 28.0 | Helped |
| Use Template H descriptors with conditional context prompt (ablation) | VLDet and RegionCLIP · OV-COCO novel categories | AP: 32.0 -> 33.8; 26.8 -> 27.6 | Helped |
| Combine Templates H and C with conditional context prompt (ablation) | VLDet and RegionCLIP · OV-COCO novel categories | AP: 32.0 -> 34.6; 26.8 -> 28.4 | Helped |
| Replace iterative LLM interaction with a static knowledge base (ablation) | VLDet and RegionCLIP · OV-COCO novel categories | AP: 34.6 -> 33.2; 28.4 -> 27.6 | Hurt |
| Add DVDet to VLDet | VLDet · PASCAL VOC test (COCO-trained) | mAP: 61.7 -> 64.0 | Helped |
| Add DVDet to VLDet | VLDet · LVIS validation (COCO-trained) | mAP: 10.0 -> 12.1 | Helped |
Surprising findings
- Template using confusing categories slightly outperforms the template using high-frequency descriptors, and combining both is best.
Limitations
- Interaction with LLMs during training: the specific LLM used for descriptor generation is not named in the main text.
- Gains on novel classes are modest for some base detectors (e.g., ~1.3% on Swin-B LVIS novel categories).
Experimental setup
Datasets: COCO LVIS PASCAL VOC
Benchmarks: OV-COCO OV-LVIS PASCAL VOC
Metrics: mAP mAP at IoU 0.5 mask AP
Baselines: ViLD Detic RegionCLIP VLDet BARON DetPro
Builds on: ViLD VLDet Detic RegionCLIP CaFo VPT
Closest papers by meaning
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.91)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.90)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.90)
- DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.89)
- Comprehensive Multi-Modal Prototypes Are Simple and Effective Classifiers for Vast-Vocabulary Object Detection (AAAI 2025 · similarity 0.89)
- Simple Image-Level Classification Improves Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.