Research map / Detection and segmentation / Open-set and few-shot detection
COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shifts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces COUNTS, a large-scale real-world dataset with 14 natural domain shifts and object-level boxes, and two benchmarks (O(OD)^2 for object detectors and OODG for MLLM grounding under in-context distribution shifts).
Problem
Object detectors and MLLMs degrade under distribution shifts, but existing OOD benchmarks focus on image classification and lack fine-grained object-level annotations for detection and grounding.
Approach
A real-world OOD dataset with domain labels and object-level boxes that supports controlled train/test domain splits for detectors and ICL-based distribution shifts (covariate, label, spurious) for MLLM grounding evaluation.
Key results
- In OODG visual grounding, GPT-4o and Gemini-1.5 reach 56.7% and 28.0% accuracy under distribution shift settings as stated in the abstract.
- DINO and DINOv2 detectors achieve the highest average OOD mAP (0.213) on O(OD)^2, significantly above Faster R-CNN and RetinaNet variants.
- Under covariate and label shifts in ICE, Gemini visual grounding average accuracy drops from a maximum of 57.0% to 28.0% (a 50.88% relative decrease), while GPT-4o declines by at most 12.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DINO or DINO V2 instead of the standard detector configurations | Object detectors · COUNTS O(OD)2, Avg | mAP: 0.135 -> 0.213; 0.156 -> 0.213 | Helped |
| Use a stronger backbone (RN-101 rather than RN-50) (ablation) | Faster R-CNN · COUNTS O(OD)2, Avg | mAP: 0.135 -> 0.148 | Helped |
| Replace FPN with NAS-FPN (ablation) | Faster R-CNN · COUNTS O(OD)2, Avg | mAP: 0.135 -> 0.136 | No clear effect |
| Replace the Standard head with SABL (ablation) | Faster R-CNN · COUNTS O(OD)2, Avg | mAP: 0.135 -> 0.162 | Helped |
| Replace FPN with NAS-FPN (ablation) | RetinaNet · COUNTS O(OD)2, Avg | mAP: 0.156 -> 0.133 | Hurt |
| Replace the Standard head with SABL (ablation) | RetinaNet · COUNTS O(OD)2, Avg | mAP: 0.156 -> 0.161 | Helped |
| Replace the Standard head with FreeAnchor (ablation) | RetinaNet · COUNTS O(OD)2, Avg | mAP: 0.156 -> 0.131 | Hurt |
| Use RN-101 rather than RN-50 (ablation) | RetinaNet · COUNTS O(OD)2, Avg | mAP: 0.156 -> 0.162 | Helped |
| Use Sup_timm rather than traditional supervised ImageNet pre-training (ablation) | Faster R-CNN · COUNTS O(OD)2, Avg | mAP: 0.141 -> 0.167 | Helped |
| Use Sup_timm rather than traditional supervised ImageNet pre-training (ablation) | RetinaNet · COUNTS O(OD)2, Avg | mAP: 0.145 -> 0.170 | Helped |
| Use MoCov2 rather than supervised ImageNet-1k pre-training (ablation) | Faster R-CNN · COUNTS O(OD)2, Avg | mAP: 0.141 -> 0.136 | Hurt |
| Use ImageNet-21k rather than ImageNet-1k pre-training (ablation) | Faster R-CNN · COUNTS O(OD)2, Avg | mAP: 0.141 -> 0.146 | Helped |
Surprising findings
- Improvements in IID performance do not necessarily translate to better OOD generalization for detectors; stronger NAS-FPN and FreeAnchor heads weaken cross-domain generalization.
- Self-supervised pre-training and larger pre-training data do not significantly improve detector OOD generalization, while training methodology (Sup_timm) gives larger gains.
Limitations
- Fine-tuning of MLLMs on COUNTS is noted but not the focus, so MLLM training-side generalization is not studied
- Visual grounding evaluated as multiple-choice selection and recognition on a limited set of models (GPT-4o, Gemini-1.5-Flash, GLaMM, Qwen2-VL)
Experimental setup
Datasets: COUNTS Open Images Visual Genome RefCOCO RefCOCO+ RefCOCOg Flickr30K GranD ImageNet-1k ImageNet-21k
Benchmarks: COUNTS O(OD)^2 COUNTS OODG
Metrics: mAP accuracy
Baselines: Faster R-CNN RetinaNet YOLOv9 DETR DINO GLaMM Qwen2-VL
Builds on: COCO-O COCO-C DomainNet
Closest papers by meaning
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.91)
- RUNA: Object-Level Out-of-Distribution Detection via Regional Uncertainty Alignment of Multimodal Representations (AAAI 2025 · similarity 0.90)
- CountGD: Multi-Modal Open-World Counting (NeurIPS 2024 · similarity 0.89)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty (NeurIPS 2026 · similarity 0.89)
- Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.