Research map / Detection and segmentation / Open-set and few-shot detection
InstructDET: Diversifying Referring Object Detection with Generalized Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds InDET, a 3.6M-instruction referring object detection dataset by prompting LLaVA and Vicuna-13B with object boxes and filtering with CLIP, then trains a DETR-style DROD model that beats prior visual grounding methods on RefCOCO/g/+ and InDET.
Problem
Existing referring expression comprehension datasets have few, limited expressions per image and do not cover diverse user detection intentions, including multi-object and abstract summarized instructions.
Approach
Use foundation models (LLaVA for image/object description, Vicuna-13B for instruction generation) guided by text prompts, in-context examples and object boxes to auto-generate diversified single- and multi-object instructions, filter them with CLIP using visual prompting and a local-vs-global score, and train a DETR-based referring object detector on the result.
Key results
- DROD (ResNet50) reaches 62.24 AP on InDET versus 43.37 for UNINEXT, and 66.90 AP with ViT-H.
- On RefCOCO val, DROD (ResNet50) scores 88.92 versus 87.64 for UNINEXT; on RefCOCOg val-u it scores 83.01 versus 80.96.
- Shuffling InDET test words drops DROD from 62.24 to 53.78 AP, a larger drop than UNINEXT (43.37 to 37.61), which the authors read as better instruction comprehension.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train DROD with InDET instead of the baseline training data | DROD · InDET test | AP: 43.37 -> 62.24 | Helped |
| Use a ViT-H backbone instead of ResNet50 | DROD · InDET test | AP: 62.24 -> 66.90 | Helped |
| Train DROD with InDET instead of the baseline training data | DROD · RefCOCO val | AP: 87.64 -> 88.92 | Helped |
| Train DROD with InDET instead of the baseline training data | DROD · RefCOCO testA | AP: 90.35 -> 90.86 | Helped |
| Train DROD with InDET instead of the baseline training data | DROD · RefCOCO testB | AP: 83.49 -> 85.57 | Helped |
| Train DROD with InDET instead of the baseline training data | DROD · RefCOCO+ val | AP: 78.14 -> 78.27 | Helped |
| Train DROD with InDET instead of the baseline training data | DROD · RefCOCO+ testA | AP: 83.22 -> 83.39 | Helped |
| Train DROD with InDET instead of the baseline training data | DROD · RefCOCO+ testB | AP: 68.71 -> 71.04 | Helped |
| Train DROD with InDET instead of the baseline training data | DROD · RefCOCOg val-u | AP: 80.96 -> 83.01 | Helped |
| Train DROD with InDET instead of the baseline training data | DROD · RefCOCOg test-u | AP: 81.86 -> 82.91 | Helped |
| Shuffle the words in each instruction | DROD · InDET test | AP: 62.24 -> 53.78 | Hurt |
| Shuffle the words in each instruction | UNINEXT · InDET test | AP: 43.37 -> 37.61 | Hurt |
Surprising findings
- Local-prompt LLaVA initialized from MiniGPT-4 tended to describe the whole image rather than the boxed object, and CLIP scores favored global descriptions, requiring a local-minus-global correction.
Limitations
- Instructions are generated from images with preset object boxes, so quality depends on foundation model hallucination, which is only partly filtered.
- The InDET test set is built with the same generation pipeline as training, so evaluation may favor models that match generated styles.
- Multi-object instructions are produced via clustering and LLM summarization, with limited human verification described.
Experimental setup
Datasets: InDET RefCOCO RefCOCO+ RefCOCOg Flickr30K MSCOCO Objects365 Visual Genome OpenImages
Benchmarks: InDET test set RefCOCO RefCOCO+ RefCOCOg
Metrics: average precision (AP) at IoU 0.5 AP by instruction group
Baselines: MDETR Grounding-DINO UNINEXT RefTR SeqTR
Builds on: DETR LLaVA MiniGPT-4 CLIP Segment Anything Model
Closest papers by meaning
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.87)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.87)
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.87)
- Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection (NeurIPS 2025 · similarity 0.86)
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.86)
- GCD: Advancing Vision-Language Models for Incremental Object Detection via Global Alignment and Correspondence Distillation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.