Research map / Detection and segmentation / Open-set and few-shot detection
DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DitHub, a version-control-inspired library of class-specific LoRA modules (per-class A matrices with a shared B) for incremental open-vocabulary object detection on Grounding DINO, achieving state-of-the-art on ODinW-13 and a new ODinW-O benchmark.
Problem
Adapting open-vocabulary object detectors to rare classes and recurring classes across domains in an incremental setting, while preserving zero-shot ability, without the forgetting and interference caused by monolithic single-weight adaptation.
Approach
Manage class-specific LoRA expert modules as branches in a version-control-like library, using a warmup phase, stochastic per-image expert selection, fetch-and-merge initialization of class A matrices, and a single shared B matrix updated by weighted merging across tasks.
Key results
- On ODinW-13 full-shot, DitHub reaches 62.19 average mAP vs 57.98 for ZiRa (+4.21) and 47.01 ZCOCO vs 46.26.
- On ODinW-O, DitHub achieves 62.38 average mAP vs 57.63 for ZiRa (+4.75).
- With LoRA rank r=2, DitHub is within 2 mAP of the best r=16 model while reducing per-class memory by a factor of eight.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DitHub instead of ZiRA | Grounding DINO Tiny · ODinW-13 Full | Avg mAP: 57.98 -> 62.19 | Helped |
| Use DitHub instead of ZiRA | Grounding DINO Tiny · ODinW-O | Avg mAP: 57.63 -> 62.38 | Helped |
| Add class-specific stochastic specialization (ablation) | DitHub · ODinW-13 | Avg mAP: 60.96 -> 62.19 | Helped |
| Add a warmup phase (ablation) | DitHub baseline fine-tuning · ODinW-13 | harmonic mean of Avg and ZCOCO mAP | Helped |
| Add A-matrix merging to retain class-specific knowledge (ablation) | Warmup-augmented DitHub · ODinW-13 | harmonic mean of Avg and ZCOCO mAP: +6.13 mAP | Helped |
| Share the B matrix across classes instead of using class-specific B matrices (ablation) | DitHub · ODinW-13 | Avg mAP: 63.22 -> 62.19 | Hurt |
| Share the B matrix across classes instead of using class-specific B matrices (ablation) | DitHub · ODinW-13 | ZCOCO mAP: 46.66 -> 47.01 | Helped |
| Reduce LoRA rank from 16 to 2 (ablation) | DitHub · ODinW-13 | Avg mAP: 62.19 -> 60.25 | Hurt |
| Adapt only the encoder rather than only the decoder (ablation) | DitHub · ODinW-13 | Avg mAP: 54.35 -> 62.19 | Helped |
| Adapt only the encoder rather than both encoder and decoder (ablation) | DitHub · ODinW-13 | Avg mAP: 59.92 -> 62.19 | Helped |
| Increase the A-matrix merging coefficient (ablation) | DitHub · ODinW-O | harmonic mean of Avg and ZCOCO mAP | Hurt |
| Increase the B-matrix merging coefficient up to 0.7 (ablation) | DitHub · ODinW-13 and ODinW-O | harmonic mean of Avg and ZCOCO mAP | Helped |
Surprising findings
- The Base fine-tuning baseline scores higher on ZCOCO than the original zero-shot Grounding DINO, attributed to catastrophic forgetting aligning the model to COCO.
- Some tasks show positive forgetting for DitHub, attributed to the B matrix initialization on a very small first task.
Limitations
- Training each run takes about 8 hours on a single GPU, and results depend on task order and seed.
- Merging coefficients lambda_A and lambda_B are dataset-dependent (0.3 for ODinW-13 vs 0.1 for ODinW-O), requiring tuning.
- Shared B matrix remains susceptible to interference and catastrophic forgetting.
Experimental setup
Datasets: ODinW-13 ODinW-35 MS COCO O365 GoldG Cap4M AODRaw
Benchmarks: ODinW-13 ODinW-O MS COCO (ZCOCO zero-shot) AODRaw
Metrics: mAP harmonic mean forgetting
Baselines: Grounding DINO ZiRa iDETR AT CL-DETR OW-DETR TFA
Builds on: LoRA Grounding DINO ZiRa
Closest papers by meaning
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.90)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.90)
- CAKE: Category Aware Knowledge Extraction for Open-Vocabulary Object Detection (AAAI 2025 · similarity 0.89)
- SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection (CVPR 2024 · similarity 0.89)
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.89)
- Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.