Research map / Detection and segmentation / Open-set and few-shot detection
Generative Region-Language Pretraining for Open-Ended Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates object detection as generative open-ended detection, pairing a class-agnostic Deformable DETR with an encoder-decoder language model (FlanT5-base) trained end-to-end on Visual Genome and GRIT image-text pseudo-labels, so object names are generated without predefined categories at inference.
Problem
Open-vocabulary object detection still requires predefined category names at inference, which is impractical when test-image categories are unknown or the label set is incomplete; the paper targets detecting dense objects and generating their names free-form.
Approach
Replace the classifier head with a language model that translates each matched object query into a free-form name, trained end-to-end with a class-agnostic DETR, language modeling loss, and a CLIP-based region-word alignment loss; pseudo-labels from the model's own beam search enrich vocabulary. Evaluation uses text-encoder similarity (CLIP/BERT) and METEOR to score generated names against ground
Key results
- On zero-shot LVIS, GenerateU (Swin-T, VG+GRIT5M) reaches 26.8 AP versus 26.0 for GLIP (Swin-T, O365+GoldG+CAP4M), without category names at inference.
- Swin-L backbone raises GenerateU to 27.9 AP on LVIS (APr 22.3).
- Training end-to-end with FlanT5-base LLM gives 26.8 AP versus 14.3 AP for a frozen MLLM (Kosmos-2) with trainable detection head only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GRIT image-text training data to VG training | GenerateU with Swin-T · LVIS minival | AP: 25.4 -> 26.8 | Helped |
| Add GRIT image-text training data to VG training | GenerateU · COCO val | AP: 33.0 -> 33.6 | Helped |
| Add GRIT image-text training data to VG training | GenerateU · Objects365 val | AP: 10.1 -> 10.5 | Helped |
| Scale the backbone from Swin-T to Swin-L | GenerateU trained on VG and GRIT5M · LVIS minival | AP: 26.8 -> 27.9 | Helped |
| Train the image encoder instead of freezing it (ablation) | GenerateU initialized from KOSMOS-2 · LVIS minival | AP: 14.3 -> 21.9 | Helped |
| Train the language model end-to-end instead of freezing it (ablation) | GenerateU initialized from FlanT5-base · LVIS minival | AP: 22.9 -> 26.8 | Helped |
| Add region-word alignment loss (ablation) | GenerateU trained on VG · LVIS minival | AP: 22.5 -> 25.4 | Helped |
| Add region-word alignment loss (ablation) | GenerateU trained on VG and GRIT · LVIS minival | AP: 25.7 -> 26.8 | Helped |
| Increase beam size from 1 to 3 (ablation) | GenerateU · LVIS minival | AP: 23.9 -> 26.8 | Helped |
| Increase beam size from 1 to 3 (ablation) | GenerateU · LVIS minival | APr: 12.3 -> 20.0 | Helped |
| Increase beam size from 3 to 6 (ablation) | GenerateU · LVIS minival | AP: 26.8 -> 26.6 | Hurt |
| Use a frozen MLLM with only the detection head trainable (ablation) | GenerateU initialized from KOSMOS-2 · LVIS minival | AP: 26.8 vs 14.3 | Hurt |
Surprising findings
- Directly transferring a frozen multimodal LLM (Kosmos-2) to open-ended detection performs poorly (14.3 AP), attributed to 224x224 pretraining resolution and image-level supervision.
- Removing the region-word alignment loss only lowers AP by 0.9 to 2.9 points, and the model with VG alone already performs competitively on zero-shot LVIS.
Limitations
- Naive pseudo-labeling is limited; supplementing missing objects is constrained by the model's own generation quality.
- Evaluation of free-form names depends on fixed external text encoders (CLIP/BERT) or METEOR, which are imperfect proxies for correctness.
- Training data scale effects are not fully explored.
Experimental setup
Datasets: Visual Genome GRIT COYO700M LVIS COCO Objects365
Benchmarks: LVIS COCO Objects365
Metrics: zero-shot fixed AP APr APc APf AP METEOR CLIP text-encoder similarity AP BERT similarity AP
Baselines: MDETR MaskRCNN Deformable DETR GLIP GLIP (C) Kosmos-2 GenerateU frozen MLLM variant
Builds on: Deformable DETR GLIP Kosmos-2 FlanT5 CLIP
Closest papers by meaning
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.92)
- DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection (CVPR 2024 · similarity 0.91)
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.90)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.89)
- Generating Enhanced Negatives for Training Language-Based Object Detectors (CVPR 2024 · similarity 0.89)
- The Devil is in the Fine-Grained Details: Evaluating Open-Vocabulary Object Detectors for Fine-Grained Understanding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.