Research map / Detection and segmentation / Open-set and few-shot detection
Open-Det: An Efficient Learning Framework for Open-Ended Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Open-Det, an efficient generative open-ended object detection framework that trains on 0.077M VG images with fewer epochs and GPUs than GenerateU while improving LVIS zero-shot AP.
Problem
Open-ended object detection (OED) models such as GenerateU need large-scale data, many GPUs, and converge slowly, with limited detection performance.
Approach
Combines a decoupled one-to-many/one-to-one decoder, a bidirectional vision-language aligner, a VLM-to-LLM prompt distillation module producing VL-prompts, a LoRA head with frozen LLM head and text denoising, and Masked Alignment and Joint losses to correct contradictory supervision and improve query classification.
Key results
- Open-Det with Swin-Tiny on VG only (0.077M images, 31 epochs) reaches 27.0 AP and 21.0 AP_r on zero-shot LVIS MiniVal, versus GenerateU's 26.8 AP and 20.0 AP_r trained on VG+GRIT5M (5.077M images).
- With Swin-Large, Open-Det reaches 33.1 AP and 31.2 AP_r versus GenerateU-Swin-Large's 27.9 AP and 22.3 AP_r.
- Open-Det uses 20.8% of GenerateU's training epochs (31 vs 149) and 4 V100 GPUs instead of 16 A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GenerateU with Open-Det, using VG training data | Swin-Tiny OED · LVIS MiniVal | AP_r: 17.4 -> 21.0 (+3.6%) | Helped |
| Replace GenerateU with Open-Det, using VG training data | Swin-Tiny OED · LVIS MiniVal | AP: 25.4 -> 27.0 (+1.6%) | Helped |
| Train Open-Det for 50 instead of 31 epochs | Open-Det Swin-Tiny · LVIS MiniVal | AP_r: +0.9% | Helped |
| Replace GenerateU with Open-Det | Swin-Large OED · COCO2017 zero-shot | AP: 33.0 -> 35.8 (+2.2%) | Helped |
| Replace GenerateU with Open-Det | Swin-Large OED · Objects365 zero-shot | AP: 10.1 -> 13.8 (+3.3%) | Helped |
| Add ODR to the detector (ablation) | Open-Det · LVIS MiniVal | AP_r: 10.2 -> 13.9 | Helped |
| Add BVLA-M (ablation) | Open-Det with ODR · LVIS MiniVal | AP_r: 13.9 -> 14.7 | Helped |
| Add VLD-M (ablation) | Open-Det with ODR and BVLA-M · LVIS MiniVal | AP: +2.8% | Helped |
| Add ONG with LoRa Head and Text Denoising (ablation) | Open-Det with VLD-M · LVIS MiniVal | AP_r: 16.3 -> 16.9 | Helped |
| Add Masked Alignment Loss and Joint Loss (ablation) | Open-Det with ONG · LVIS MiniVal | AP_r: +4.1% | Helped |
| Add Masked Alignment Loss (ablation) | Open-Det loss ablation · LVIS MiniVal | AP_r: 16.9 -> 18.6 (+1.7%) | Helped |
| Add Joint Loss (ablation) | Open-Det loss ablation · LVIS MiniVal | AP_r: 16.9 -> 19.5 (+2.6%) | Helped |
Surprising findings
- Freezing the LLM weights yields much lower performance (AP 20.0) than training the LLM (AP 27.0) for Open-Det, suggesting a large domain shift between pre-training and region-text alignment.
Limitations
- Performance constrained by cross-modal semantic discrepancies between visual regions and image-like text embeddings.
- Gains over GenerateU on COCO and Objects365 are evaluated only with Swin-Large and rely on zero-shot transfer from VG.
Experimental setup
Datasets: Visual Genome LVIS
Benchmarks: LVIS MiniVal COCO2017 Objects365
Metrics: AP_r AP_c AP_f AP
Baselines: GenerateU GLIP Grounding DINO MDETR MaskRCNN Deformable DETR
Builds on: GenerateU DINO CLIP FlanT5 DETR
Closest papers by meaning
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.92)
- DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection (CVPR 2024 · similarity 0.91)
- DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.90)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.90)
- Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (ICLR 2025 · similarity 0.90)
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.