Research map / Detection and segmentation / Open-set and few-shot detection
LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Co-trains an open-vocabulary detector (MM Grounding DINO) with an LLM that generates image-level detailed captions and region-level phrases on a new 1.12M-sample GroundingCap-1M dataset, improving zero-shot LVIS and other benchmarks.
Problem
Open-vocabulary object detectors are trained mainly with region-level annotations and short phrases, lacking image-level detailed supervision that could improve rare-class and generalization performance.
Approach
Append an LLM to a DETR-based open-vocabulary detector and train it jointly with grounding loss plus image-level long-caption generation loss and region-level phrase generation loss, using a projector pre-alignment step; the LLM is discarded at inference.
Key results
- On LVIS minival, LLMDet with Swin-T outperforms MM-GDINO baseline by 3.3 AP and 3.1 APr (44.7 AP vs 41.4); Swin-L variant reaches 50.6 AP per text (51.1 in table).
- On ODinW35, LLMDet (Swin-T) achieves 23.8 AP vs 23.1 for MM-GDINO.
- Ablation: adding image-level plus region-level generation on top of grounding data raises APr from 34.2 to 37.3 on LVIS minival (Swin-T).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| co-train with image-level detailed caption generation and region-level phrase generation | MM-GDINO Swin-T · LVIS minival | AP: 41.4 -> 44.7 | Helped |
| co-train with image-level detailed caption generation and region-level phrase generation | MM-GDINO Swin-T · LVIS minival | APr: 34.2 -> 37.3 | Helped |
| co-train with image-level detailed caption generation and region-level phrase generation | MM-GDINO Swin-T · ODinW35 | AP: 23.1 -> 23.8 | Helped |
| co-train with image-level detailed caption generation and region-level phrase generation | MM-GDINO Swin-T · COCO-O | AP: 34.0 -> 36.1 | Helped |
| co-train with image-level detailed caption generation and region-level phrase generation | MM-GDINO Swin-T · RefCOCO val | Top-1 accuracy: 66.0 -> 69.5 | Helped |
| add image-level caption generation alongside grounding (ablation) | LLMDet · LVIS minival | AP: 43.8 -> 44.0 | Helped |
| add region-level generation alongside image-level generation (ablation) | LLMDet · LVIS minival | APr: 33.4 -> 37.3 | Helped |
| replace GroundingCap-1M captions with LLaVA-OneVision-7B generated captions (ablation) | LLMDet · LVIS minival | APr: 38.6 -> 33.5 | Hurt |
| replace detailed captions with original captions and short grounding texts (ablation) | LLMDet · LVIS minival | AP: 43.6 -> 43.2 | Hurt |
| remove the LCS dataset from GroundingCap-1M (ablation) | LLMDet · LVIS minival | AP: 44.5 -> 42.8 | Hurt |
| retain speculative sentences during caption post-processing (ablation) | LLMDet · LVIS minival | APr: 38.6 -> 35.0 | Hurt |
| add cross-attention to region-level generation (ablation) | LLMDet · LVIS minival | AP: 44.0 -> 44.5 | Helped |
Surprising findings
- Region-level captioning alone does not improve performance, since region captions are just class names or phrases carrying no extra information.
- Captions generated by LLaVA-OneVision-7B instead of Qwen2-VL-72B reduce APr by 5.1 points, showing caption quality matters for rare classes.
Limitations
- Dataset is only about 1M samples, much smaller than datasets used by other open-vocabulary detectors, limiting performance.
- Captions depend on a 72B proprietary-quality generator (Qwen2-VL-72B), and residual hallucinations remain.
- Only evaluated on the MM-GDINO baseline; applicability to other detectors like DetCLIP is untested.
Experimental setup
Datasets: GroundingCap-1M COCO V3Det GoldG LCS-558k Objects365 LVIS ODinW COCO-O RefCOCO RefCOCO+ RefCOCOg
Benchmarks: LVIS minival LVIS val ODinW13 ODinW35 COCO-O RefCOCO RefCOCO+ RefCOCOg gRefCOCO
Metrics: fixed AP APr AP Top-1 accuracy Pr(F1=1, IoU>=0.5) N-acc detailedness score hallucination score
Baselines: MM-GDINO GLIP GLIPv2 CapDet DetCLIPv3 Grounding-DINO OWL-ST T-Rex2 DetCLIPv2 OV-DINO
Builds on: MM Grounding DINO GLIP LLaVA-OneVision DETR
Closest papers by meaning
- DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection (CVPR 2024 · similarity 0.90)
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.90)
- Language-conditioned Detection Transformer (CVPR 2024 · similarity 0.89)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.88)
- OVMR: Open-Vocabulary Recognition with Multi-Modal References (CVPR 2024 · similarity 0.88)
- Open-Vocabulary Object Detection via Language Hierarchy (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.