Research map / Detection and segmentation / Open-set and few-shot detection
Language-conditioned Detection Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DECOLA, a language-conditioned Deformable DETR that generates accurate offline pseudo-boxes for image-level-tagged ImageNet-21K data, then trains an open-vocabulary detector on them, reaching state of the art on open-vocabulary and zero-shot LVIS and cross-dataset benchmarks.
Problem
Open-vocabulary object detection needs detection-level supervision, but image-level weakly labeled data yields poor pseudo-boxes for unseen classes because standard open-vocabulary detectors apply image-level labels only after class-agnostic proposals, leading to low coverage.
Approach
Make the DETR query selection itself conditioned on each queried class's text embedding (similarity-based objectness per class, with per-class isolated self-attention), so the detector focuses proposals on the given concepts; use it as an offline pseudo-labeler for weakly labeled images and then train a standard open-vocabulary detector on those labels.
Key results
- Open-vocabulary LVIS with Swin-L: DECOLA reaches 10.4 APrare improvement over baseline + self-train (46.9 APbox novel) and 3.6 mAP gain.
- Direct zero-shot LVIS v1.0 val with Swin-T: DECOLA gets 27.2 APrare, outperforming prior best by 17.1 APrare and 9.4 mAP.
- Standard LVIS with ResNet-50 CenterNet2: DECOLA labels improve Detic's APbox rare from 31.4 to 35.6 (+4.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| language-conditioned pseudo-labeling (DECOLA Phase 1 labels) instead of standard self-training | ResNet-50 open-vocabulary LVIS · Open-vocabulary LVIS | APbox novel: +4.4 over baseline + self-train (23.2 to 27.6) | Helped |
| language-conditioned pseudo-labeling instead of standard self-training | Swin-L open-vocabulary LVIS · Open-vocabulary LVIS | mAPbox: +3.6 (51.8 to 55.2) | Helped |
| language-conditioning of query selection and pseudo-labeling | Standard LVIS DETR (ResNet-50, Swin-B, Swin-L) · Standard LVIS | APbox rare: +5.9 / +5.4 / +6.2 over baseline + self-train | Helped |
| DECOLA pseudo-labels for CenterNet2 (Detic-base) on R-CNN detectors | Detic with ResNet-50 and Swin-B · Standard LVIS CenterNet2 | APbox rare: +4.2 (ResNet-50), +1.8 (Swin-B) over Detic | Helped |
| DECOLA Phase 2 open-vocabulary detector vs Detic on cross-dataset transfer | ResNet-50 zero-shot transfer · COCO, Object365, OpenImages | AP / APflat50: COCO AP +0.5 (39.8 to 40.3); O365 AP 14.5 to 15.0; OI APflat50 41.6 to 43.3 | Helped |
| online pseudo-labeling (baseline + self-train) vs offline DECOLA Phase 1 labeling (ablation) | Unseen-class pseudo-labels · ImageNet-21K pseudo-labeling c-AP@20 | c-AP: online self-labeling suffers sharp early drop and underperforms offline DECOLA after full iterations | Hurt |
| removing language conditioning from first-stage query selection (baseline pseudo-labeler) (ablation) | Unseen-class pseudo-labeling · ImageNet-21K c-AR | c-AR first and second stage: significant c-AR improvement on both proposals and predictions with conditioning | Helped |
| memory-efficient isolated self-attention and top-K=10,000 pixel matching in Phase 1 | DECOLA Phase 1 training · Training cost | training time and memory overhead: increases training time and memory by less than 20% over baselines | No clear effect |
Surprising findings
- Online pseudo-labeling (Detic-style) of baseline+self-train degrades frequent-class performance and shows sharp early drops, while offline DECOLA pseudo-labeling is more stable.
Limitations
- Relies on Phase 1 pseudo-label quality; mixed gains depend on backbone and dataset.
- Scoring is per-class with n=300 queries per class and top-K pixel restriction, adding memory and time overhead for many classes.
Experimental setup
Datasets: LVIS ImageNet-21K Objects365 COCO OpenImages Visual Genome
Benchmarks: LVIS LVIS minival LVIS v1.0 val COCO Object365 OpenImages
Metrics: APbox novel APrare APc APf mAP AP AP50 AP75 APflat50 c-AP
Baselines: OV-DETR OWLv2 Detic baseline + self-train MDETR GLIP GroundingDINO MQ-GLIP MQ-GroundingDINO DITO ViLD BARON
Builds on: Deformable DETR Detic CLIP OV-DETR Federated Loss
Closest papers by meaning
- Taming Self-Training for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.89)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.89)
- Open-Vocabulary Object Detection via Language Hierarchy (NeurIPS 2024 · similarity 0.89)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.88)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.88)
- DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.