Research map / Vision-language and multimodal / CLIP and zero-shot
Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Roboflow100-VL, a 100-dataset multi-domain object detection benchmark evaluating VLMs and specialist detectors in zero-shot, few-shot, semi-supervised and fully-supervised settings, with multi-modal annotator instructions.
Problem
Vision-language models achieve strong zero-shot detection on common objects but fail on out-of-distribution concepts, tasks, and imaging modalities; existing benchmarks (RefCOCO, ODinW) do not measure this well.
Approach
A benchmark of 100 Roboflow Universe datasets with out-of-distribution concepts, each paired with multi-modal annotator instructions (few visual examples plus rich text descriptions), to test few-shot concept alignment of detectors and MLLMs.
Key results
- GroundingDINO achieves 15.7 mAP zero-shot on RF100-VL versus 49.2 on ODinW-13, and 33.6 mAP with 10-shot fine-tuning.
- Gemini 2.5 Pro and Qwen2.5-VL (72B) reach only about 8% zero-shot mAP with class names, and GroundingDINO and Gemini reach under 2% on some medical datasets.
- The winning team of the CVPR 2025 Foundational FSOD challenge reached 50.4 mAP, beating the best few-shot baseline by about 17 mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune GroundingDINO on 10 shots rather than use it zero-shot | GroundingDINO · Roboflow100-VL, All | mAP: 15.7 -> 33.6 | Helped |
| Use instructions instead of class names (ablation) | Qwen 2.5 VL (72B) · Roboflow100-VL, zero-shot, All | mAP: 5.6 -> 7.6 | Helped |
| Use instructions instead of class names (ablation) | Gemini 2.5 Pro · Roboflow100-VL, zero-shot, All | mAP: 11.6 -> 4.5 | Hurt |
| Add images to class-name prompts (ablation) | MQ-GLIP · Roboflow100-VL, 10-shot, All | mAP: 12.2 -> 12.2 | No clear effect |
| Use class names and images instead of images alone (ablation) | MQ-GLIP · Roboflow100-VL, 10-shot, All | mAP: 6.4 -> 12.2 | Helped |
| Add Federated Loss (ablation) | YOLOv8n · Roboflow100-VL, 10-shot, All | mAP: 20.2 -> 21.7 | Helped |
| Add Federated Loss (ablation) | YOLOv8s · Roboflow100-VL, 10-shot, All | mAP: 20.7 -> 23.6 | Helped |
| Add Federated Loss (ablation) | YOLOv8m · Roboflow100-VL, 10-shot, All | mAP: 20.3 -> 22.6 | Helped |
| Add STAC pseudo-labeling (ablation) | YOLOv8n · Roboflow100-VL, 10% labeled data, All | mAP: 39.3 -> 41.2 | Helped |
| Add STAC pseudo-labeling (ablation) | YOLOv8s · Roboflow100-VL, 10% labeled data, All | mAP: 42.3 -> 43.1 | Helped |
| Add STAC pseudo-labeling (ablation) | YOLOv8m · Roboflow100-VL, 10% labeled data, All | mAP: 42.5 -> 43.3 | Helped |
| Select few-shot examples using largest average bounding-box area instead of random selection (ablation) | Detic with Federated Fine-Tuning · nuImages, 5-shot, All | AP: 16.58 -> 16.94 | Helped |
Surprising findings
- Larger MLLMs such as Qwen2.5-VL 72B underperform smaller GroundingDINO in zero-shot detection.
- Newer YOLO versions do not improve over YOLOv8 on RF100-VL despite COCO gains, suggesting overfitting to COCO.
Limitations
- Annotations come from crowdsourced Roboflow Universe datasets with uncertain quality, especially in specialized medical domains.
- Annotator instructions are generated by GPT-4o and manually verified, and may not match real-world instructions.
- Generalist MLLMs do not report per-box confidence, so evaluation relies on default scores and no NMS.
Experimental setup
Datasets: RF100-VL (100 Roboflow Universe datasets) COCO OdinW-13 nuImages
Benchmarks: Roboflow100-VL COCO Val OdinW-13
Metrics: mAP AP mAP 50:95
Baselines: Detic GroundingDINO OWLv2 MQ-GLIP Qwen2.5-VL (72B) Gemini 2.5 Pro YOLOv8 YOLOv11 LW-DETR
Builds on: GroundingDINO MQ-Det Detic STAC Roboflow100
Closest papers by meaning
- Revisiting Few-Shot Object Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.94)
- ROD-MLLM: Towards More Reliable Object Detection in Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.89)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.89)
- COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shifts (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.