Research map / Detection and segmentation / Open-set and few-shot detection
Search and Detect: Training-Free Long Tail Object Detection via Web-Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SearchDet, a training-free long-tail object detector that retrieves positive and negative web images per label, builds an attention-weighted query with DINOv2, and combines SAM region proposals with similarity heatmaps.
Problem
Open-vocabulary object detection models underperform on long-tail (rare) concepts, and improving them typically requires costly pre-training or fine-tuning.
Approach
Use Google image search to retrieve positive and negative exemplars for a label, combine their DINOv2 embeddings via softmax-attention weighting against the input image (positive minus negative), and ground objects by matching SAM masks and a patch-similarity heatmap, with frequency-based adaptive thresholding; no training is involved.
Key results
- SearchDet reaches 59.34 mAP on COCO val2017, versus 52.2 for T-Rex2 (Text) and 48.4 for GroundingDINO-L.
- On LVIS minival rare classes, SearchDet scores 55.9 mAP_r, above T-Rex2 (Visual-G) at 43.8 and T-Rex2 (Text) at 42.7.
- In 10-shot MS-COCO, SearchDet reaches 61.4 mAP versus 52.9 for DE-ViT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add negative support images to positive support images (ablation) | SearchDet · COCO | mAP: 45.80 -> 59.34 | Helped |
| add heatmap refinement of SAM predictions (ablation) | SearchDet · COCO | mAP: 51.07 -> 59.34 | Helped |
| replace mean-pooling with attention-based pooling of support images (ablation) | SearchDet · COCO | mAP: 55.47 -> 59.34 | Helped |
| increase the number of positive and negative support images from 1 to 10 (ablation) | SearchDet · COCO-val | mAP: 49.70 -> 59.34 | Helped |
| use web-retrieved exemplars instead of the compared baseline | COCO val2017 · COCO val2017 | mAP: 52.2 -> 59.3 | Helped |
Surprising findings
- A single positive and negative support image already yields 49.70 mAP on COCO, and performance rises steadily with more retrieved images.
- SearchDet trails T-Rex2 on LVIS overall and on frequent/common subsets, which the authors attribute to the lack of training.
Limitations
- Inference is slower for unseen classes because web image scraping dominates latency (about 2.9 s).
- Performance on LVIS lags behind training-based T-Rex2 at frequent and common categories, using only 10 support images.
- Results depend on web search availability and quality; generic labels like '20' or 'boat' retrieve irrelevant images.
Experimental setup
Datasets: COCO val2017 LVIS minival ODinW-35 Roboflow-100 web images retrieved via Google (10 positive and 10 negative per class)
Benchmarks: COCO val2017 LVIS minival-1203 ODinW-35 Roboflow-100 10-shot MS-COCO
Metrics: mAP mAP at IoU 0.5 mAP rare/common/frequent
Baselines: GLIP-L DINOv GroundingDINO-L T-Rex2 (Text) T-Rex2 (Visual-G) DE-ViT Meta R-CNN CrossTransformer
Builds on: DINOv2 Segment Anything Model GroundingDINO few-shot object detection
Closest papers by meaning
- Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection (NeurIPS 2023 · similarity 0.88)
- QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection (CVPR 2022 · similarity 0.88)
- SimLTD: Simple Supervised and Semi-Supervised Long-Tailed Object Detection (CVPR 2025 · similarity 0.87)
- MatchDet: A Collaborative Framework for Image Matching and Object Detection (AAAI 2024 · similarity 0.86)
- Long-tailed Object Detection Pretraining: Dynamic Rebalancing Contrastive Learning with Dual Reconstruction (NeurIPS 2024 · similarity 0.85)
- CQ-DINO: Mitigating Gradient Dilution via Category Queries for Vast Vocabulary Object Detection (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.