Research map / Vision-language and multimodal / CLIP and zero-shot
Revisiting Few-Shot Object Detection with Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits few-shot object detection with web-scale VLMs, proposes the Foundational FSOD benchmark on repurposed nuImages where detectors are pre-trained on any data and aligned to K-shot multi-modal examples, and reports baselines plus results from a CVPR 2024 competition.
Problem
Conventional few-shot object detection (FSOD) protocols assume disjoint base/novel classes and ignore web-scale VLM pre-training, so zero-shot VLMs already outperform FSOD methods and target-concept misalignment is not addressed.
Approach
Reformulate FSOD as aligning pre-trained foundation models to target concepts using K-shot multi-modal (text + visual) examples, with no base/novel split; also propose a pseudo-negative federated loss that uses zero-shot predictions to determine negative classes per image.
Key results
- GroundingDINO zero-shot reaches 48.3 AP on COCO FSOD (30-shot setting), versus 33.1 AP for the best prior FSOD method DiGeo.
- On nuImages 10-shot, MQ-GLIP multi-modal prompting reaches 21.42 AP, the best open-source baseline; the CVPR 2024 competition winner PHP_hhh reached 45.35 AP.
- Pseudo-negative federated fine-tuning improves Detic from 16.09 to 17.24 AP on nuImages 10-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use zero-shot GroundingDINO instead of the strongest few-shot baseline DiGeo | COCO FSOD, 30-shots · COCO FSOD | AP: 33.1 -> 48.3 | Helped |
| Add multi-modal visual and text prompts rather than visual prompts alone | MQ-GLIP · nuImages 10-shot | AP (All): 14.07 -> 21.42 | Helped |
| Add multi-modal visual and text prompts rather than language prompts alone | MQ-GLIP · nuImages 10-shot | AP (All): 17.01 -> 21.42 | Helped |
| Add synonyms to class-name prompts | Detic · nuImages 10-shot | AP (All): 14.40 -> 14.92 | Helped |
| Use federated fine-tuning with pseudo-negatives instead of standard fine-tuning | Detic · nuImages 10-shot | AP (All): 16.09 -> 17.24 | Helped |
| Add chat-generated descriptions to the MQ-GLIP prompts | MQ-GLIP · nuImages 10-shot | AP (All): 21.42 -> 22.09 | Helped |
| Add large-scale pre-training and a CLIP classifier | TFA · nuImages 5-shot | AP (All): 2.02 -> 15.12 | Helped |
| Use pseudo-negatives instead of standard fine-tuning (ablation) | Detic · nuImages 10-shot | AP (All): 15.53 -> 16.67 | Helped |
| Use pseudo-negatives instead of FedLoss with sampled negatives (ablation) | Detic · nuImages 10-shot | AP (All): 15.57 -> 16.67 | Helped |
| Replace LVIS-RPN with GT-RPN (ablation) | RegionCLIP zero-shot · nuImages | AP (All): 2.34 -> 26.44 | Helped |
| Replace LVIS-RPN with Detic-RPN (ablation) | RegionCLIP zero-shot · nuImages | AP (All): 2.34 -> 3.79 | Helped |
| Filter Detic-RPN proposals below 0.5 objectness (ablation) | RegionCLIP zero-shot with Detic-RPN · nuImages | AP (All): 3.79 -> 7.64 | Helped |
Surprising findings
- Zero-shot GroundingDINO outperforms state-of-the-art few-shot detectors on COCO by a wide margin, suggesting current FSOD protocols violate the pre-training assumption.
- TFA trained on COCO-base or LVIS-base achieves under 3 AP on nuImages, while swapping to a frozen CLIP classifier raises 5-shot AP to 15.12.
Limitations
- Performance on rare categories such as stroller and debris remains considerably lower than on common classes.
- Benchmark contamination is hard to rule out since foundation models may have seen test-domain data; the paper relies on prohibiting nuImages training in the challenge.
- Results use only nouns and synonyms as text prompts rather than rich textual descriptions from annotator instructions.
Experimental setup
Datasets: nuImages COCO LVIS ImageNet-21K Objects365 GoldG Cap4M Cap24M
Benchmarks: nuImages Foundational FSOD COCO FSOD LVIS 10-shot
Metrics: AP Many/Medium/Few AP Base AP Novel AP
Baselines: GroundingDINO DiGeo TFA Detic GLIP MQ-GLIP RegionCLIP GPT-4o
Builds on: Detic federated loss MQ-GLIP TFA
Closest papers by meaning
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.94)
- GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.89)
- Few-Shot Object Detection with Foundation Models (CVPR 2024 · similarity 0.89)
- Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.88)
- Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning (CVPR 2025 · similarity 0.88)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.