Research map / Detection and segmentation / Open-set and few-shot detection
DoGA: Enhancing Grounded Object Detection via Grouped Pre-Training with Attributes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DoGA, a grounded object detection pre-training method that uses LLM-generated attribute-based definitions and hard-negative captions, with paralleled text encoding and many-to-many group-wise training on Grounding DINO.
Problem
Grounded/open-vocabulary detectors pre-trained on coarse prompts (plain category names, brief grounding phrases) degrade sharply on fine-grained, attribute-rich or contextual descriptions and hallucinate on hard-negative captions.
Approach
Builds attribute-based definitions (super-category, visual attributes) via GPT-4 and hard-negative concepts from attribute/entity vocabularies, encodes each prompt split into parallel segments, extracts only entity embeddings for fusion and loss, and trains with prompt-wise group decoding that allows many-to-many assignment to synonymous prompts.
Key results
- Zero-shot LVIS minival 38.0 AP (33.5 APr) with pre-training on O365v1, GoldG, VAW+, outperforming GLIP, Grounding DINO and DetCLIP by 10.0/9.6/1.1 AP.
- COCO val AP 50.2 with the full pre-training setup (vs. 46.7 for Grounding DINO-T on O365v1+GoldG+Cap4M setting in the table comparisons).
- On the Grounding DINO baseline with entity-dropped contextual prompts, mAP falls from 48.2 to 19.2 on COCO, motivating the attribute-based prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add contextual prompts (ablation) | DoGA · LVIS minival | AP: 24.5 -> 27.7 | Helped |
| Add contextual prompts (ablation) | DoGA · COCO | AP: 46.5 -> 46.0 | Hurt |
| Add hard-negative prompts (ablation) | DoGA with contextual prompts · LVIS minival | AP: 27.7 -> 30.6 | Helped |
| Add group-wise synonymous training (ablation) | DoGA with hard-negative prompts · LVIS minival | AP: 30.6 -> 34.2 | Helped |
| Add prompt-wise test-time augmentation (ablation) | DoGA · LVIS minival | AP: 34.2 -> 35.6 | Helped |
| Add prompt-wise test-time augmentation (ablation) | DoGA · COCO | AP: 49.8 -> 49.8 | No clear effect |
| Add VAW+ pre-training data (ablation) | DoGA · LVIS minival | AP: 35.6 -> 38.0 | Helped |
| Add VAW+ pre-training data (ablation) | DoGA · COCO | AP: 49.8 -> 50.2 | Helped |
| Use more capable LLM-generated prompts (ablation) | DoGA · LVIS minival | AP: 27.8 -> 34.2 | Helped |
| Reduce prompt noise scale (ablation) | DoGA · LVIS minival | AP: 22.0 -> 34.2 | Helped |
| Train DoGA instead of Grounding-DINO-T | Swin-T detection model · COCO val | AP: 48.4 -> 50.2 | Helped |
| Train DoGA instead of DetCLIP-T | Swin-T detection model · LVIS minival | AP: 35.9 -> 38.0 | Helped |
Surprising findings
- Adding contextual attribute definitions to plain category names degrades the entity distribution of Grounding DINO (offset and overlap across classes), with contextual prompts sharply reducing AP on COCO for the baseline.
- Performance on negative/noisy prompts degrades gracefully with LLM quality, with GPT-4-generated definitions giving 34.2 vs. 27.8 for babbage.
Limitations
- Relies on LLM-generated (GPT-4) attribute definitions whose quality affects performance; noisy prompts reduce accuracy.
- Only evaluated at Swin-Tiny/Grounding DINO-T scale; scaling to larger models left to future work.
Experimental setup
Datasets: Objects365 (O365v1) GoldG VAW+ VAW GQA Visual Genome COCO LVIS ODinW13 Flickr30k RefCOCO RefCOCO+
Benchmarks: COCO LVIS minival ODinW13 Flickr30k RefCOCO RefCOCO+ RefCOCOg
Metrics: AP APr/APc/APf APavg R@1
Baselines: GLIP-T Grounding-DINO-T DetCLIP-T DesCo-GLIP-T Group-DETR
Builds on: Grounding DINO GLIP Group-DETR DetCLIP
Closest papers by meaning
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.90)
- GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.88)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Revisiting Few-Shot Object Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.87)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.