Research map / Detection and segmentation / Open-set and few-shot detection
Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IVLOD, the task of incrementally adapting vision-language object detectors to specialized domains while keeping zero-shot generalization, and ZiRa, a reparameterizable dual-branch adapter trained with a zero-interference L1 loss on Grounding DINO and OV-DINO.
Problem
Incrementally adapting pre-trained vision-language object detectors (VLODMs) to new specialized domains causes catastrophic forgetting of both prior downstream tasks and the model's zero-shot generalization to general-domain (COCO) categories.
Approach
Insert parallel reparameterizable dual branches (a high-learning-rate branch and a low-learning-rate branch) on the language and vision sides of a frozen VLODM, merge the high-LR branch into the low-LR branch after each task, and penalize the L1 norm of the RDB output and of the high-LR branch output (Zero-interference Loss) so that the model's outputs stay close to the pre-trained model without s
Key results
- ZiRa with Grounding DINO reaches 46.06 ZCOCO and 59.73 Avg on ODinW-13 in the full-shot setting, versus 32.15 ZCOCO for CL-DETR and 37.32 for iDETR.
- With OV-DINO, ZiRa reaches 49.07 ZCOCO versus 34.52 for CL-DETR and 37.71 for iDETR in the full-shot setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZiRa instead of CL-DETR | Grounding DINO Swin-Tiny · COCO (ZCOCO), full shots | AP: 32.15 -> 46.06 | Helped |
| Use ZiRa instead of iDETR | Grounding DINO Swin-Tiny · COCO (ZCOCO), full shots | AP: 37.32 -> 46.06 | Helped |
| Use ZiRa instead of CL-DETR | OV-DINO Swin-Tiny · COCO (ZCOCO), full shots | AP: 34.52 -> 49.07 | Helped |
| Add Rep+ (differentiated branch learning rates and reparameterization) (ablation) | ZiRa · ODinW-13 | Avg AP: 58.12 -> 59.98 | Helped |
| Add both ZiL losses without Rep+ (ablation) | ZiRa · COCO | ZCOCO AP: 39.72 -> 46.09 | Helped |
| Add both ZiL losses without Rep+ (ablation) | ZiRa · ODinW-13 | Avg AP: 58.12 -> 54.24 | Hurt |
| Add Rep+ to the configuration with both ZiL losses (ablation) | ZiRa · ODinW-13 | Avg AP: 54.24 -> 59.73 | Helped |
| Add Lhlrb loss to Rep+ with Lrdb loss (ablation) | ZiRa · COCO | ZCOCO AP: 46.01 -> 46.06 | No clear effect |
| Add Lhlrb loss to Rep+ with Lrdb loss (ablation) | ZiRa · ODinW-13 | Avg AP: 58.43 -> 59.73 | Helped |
| Adapt on both vision and language sides with ZiL rather than vision only with ZiL (ablation) | ZiRa · ODinW-13 | Avg AP: 57.64 -> 59.73 | Helped |
| Use RDB instead of a single branch (ablation) | ZiRa · ODinW-13 | Avg AP: 55.07 -> 59.73 | Helped |
| Vary ZiL weight lambda from 0.00 to 1.00 (ablation) | ZiRa · COCO and ODinW-13 | hAP: 47.19; 51.46; 52.05; 52.01; 51.51; 50.14; 48.46 | Mixed |
Surprising findings
- Adding the L_rdb term alone (without reparameterization) can reduce downstream Avg, since it limits plasticity of the RDB.
- Using the L2 norm yields the best ZCOCO but lower Avg, while Smooth L1 gives the best hAP.
Limitations
- Current implementation is based on the DETR architecture and may not be optimal for all VLODMs.
- Only evaluated on the IVLOD task; generalization to other incremental learning tasks is unexplored.
Experimental setup
Datasets: COCO ODinW-13 Objects365 GoldG Cap4M Cap1M
Benchmarks: COCO ODinW-13
Metrics: ZCOCO zero-shot mAP Avg mAP on ODinW-13 hAP (harmonic mean)
Baselines: TFA OW-DETR CL-DETR iDETR Adapting-tuning (AT)
Builds on: Grounding DINO RepVGG CL-DETR iDETR Adapting-tuning
Closest papers by meaning
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.88)
- Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning (AAAI 2024 · similarity 0.87)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.87)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.87)
- GCD: Advancing Vision-Language Models for Incremental Object Detection via Global Alignment and Correspondence Distillation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.