Research map / Detection and segmentation / Open-set and few-shot detection
Scene-adaptive and Region-aware Multi-modal Prompt for Open Vocabulary Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMP, a distillation-based open-vocabulary object detector that adds scene-adaptive multi-modal prompts (shared plus low-rank scene prompts with instance-wise selection) and a region-aware alignment module, reaching state-of-the-art on OV-COCO and OV-LVIS.
Problem
Open vocabulary object detection via distillation from CLIP suffers from weak adaptation to diverse classes (single shared prompt or fixed templates) and a gap between image-level CLIP pretraining and region-level detection.
Approach
A scene-adaptive prompt generator builds a common prompt plus low-rank (Hadamard-product) scene-specific prompts, selected per instance via learnable keys; a region-aware module injects region-mask prompts for positional info and text prompts into visual features for region-level alignment.
Key results
- On OV-COCO, SAMP reaches 34.8 novel mAP50 (61.4 base, 54.2 all), above BARON's 34.0 novel mAP50.
- On OV-LVIS, SAMP reaches 27.8 APr, outperforming the best prior approach by 4.6 points.
- Ablation on OV-COCO: scene-adaptive prompt generator alone gives +5.2 novel mAP50; region-aware alignment gives +4.7 novel; combined SAMP reaches 54.2 mAP50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SAMP instead of BARON | Faster R-CNN, R50-FPN (SoCo) · OV-COCO | mAP50 (novel): 34.0 -> 34.8 | Helped |
| use SAMP instead of BARON | Faster R-CNN, R50-FPN (SoCo) · OV-COCO | mAP50 (base): 60.4 -> 61.4 | Helped |
| use SAMP instead of BARON | Faster R-CNN, R50-FPN (SoCo) · OV-COCO | mAP50: 53.5 -> 54.2 | Helped |
| use SAMP instead of BARON | Faster R-CNN, R50-FPN (SoCo) · OV-LVIS | APr: 23.2 -> 27.8 | Helped |
| use SAMP instead of BARON | Faster R-CNN, R50-FPN (SoCo) · OV-LVIS | AP: 29.5 -> 34.3 | Helped |
| add the scene-adaptive prompt generator (ablation) | OV-COCO baseline detector · OV-COCO | mAP50 (novel): 27.6 -> 32.8 | Helped |
| add region-aware multi-modal alignment (ablation) | OV-COCO baseline detector · OV-COCO | mAP50 (novel): 27.6 -> 32.3 | Helped |
| use scene-adaptive prompts instead of shared prompts in the text branch (ablation) | OV-COCO detector · OV-COCO | mAP50 (novel): 29.3 -> 31.6 | Helped |
| use scene-adaptive prompts instead of shared prompts in the visual branch (ablation) | OV-COCO detector · OV-COCO | mAP50 (novel): 28.8 -> 30.7 | Helped |
| use scene-adaptive multi-modal prompts instead of shared prompts (ablation) | OV-COCO detector · OV-COCO | mAP50 (novel): 29.1 -> 32.8 | Helped |
| use low-rank decomposition for scene prompts instead of random prompts (ablation) | OV-COCO detector · OV-COCO | mAP50 (novel): 28.4 -> 31.6 | Helped |
| add the region prompt (ablation) | OV-COCO detector with RMA · OV-COCO | mAP50 (novel): 27.6 -> 30.4 | Helped |
Surprising findings
- A learnable prompt shared across all classes can underperform hand-crafted templates for some superclasses on COCO.
Limitations
- Gains over region-aware pretraining methods on base classes are lower (BARON is 6.6 points higher on base mAP50 in the compared setting).
- Scene prompt count and size are hyperparameters tuned on OV-COCO (15) and OV-LVIS (100).
Experimental setup
Datasets: COCO LVIS v1.0 Pascal VOC
Benchmarks: OV-COCO OV-LVIS Pascal VOC
Metrics: mAP50 novel (mAP50_N) mAP50 base mAP50 all APr AP
Baselines: ViLD DetPro OADP BARON PromptDet Detic RegionCLIP F-VLM OV-RCNN MEDet VLDet RO-ViT
Builds on: ViLD DetPro OADP CoOp MaPLe VPT
Closest papers by meaning
- Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm (AAAI 2025 · similarity 0.84)
- Exploring Region-Word Alignment in Built-in Detector for Open-Vocabulary Object Detection (CVPR 2024 · similarity 0.84)
- Simple Image-Level Classification Improves Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.84)
- LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models (CVPR 2025 · similarity 0.84)
- DoGA: Enhancing Grounded Object Detection via Grouped Pre-Training with Attributes (AAAI 2025 · similarity 0.84)
- DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.