Research map / Detection and segmentation / Open-set and few-shot detection
Recognize Any Regions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RegionSpot, which keeps SAM and CLIP frozen and trains only a lightweight cross-attention module that fuses SAM position-aware box tokens with CLIP image-level feature maps for open-world region recognition.
Problem
Open-world region-level recognition (classifying given boxes/regions in unconstrained images) is costly to train and suffers from noisy region-label data and missing context when using cropped-region CLIP features.
Approach
Combine frozen localization (SAM) and frozen vision-language (CLIP) foundation models via cross-attention where SAM box-prompt tokens query the whole-image CLIP feature map, so only a small attention module is trained on detection data.
Key results
- On LVIS val with GLIP-L boxes, RegionSpot-Pro+ reaches 29.8 AP_all vs 26.9 for GLIP-L, and 30.2 vs 17.1 AP_r for rare categories.
- On LVIS minival, RegionSpot-Pro+ gets 33.2 AP_r vs 28.2 for GLIP-L and 22.2 for GroundingDINO-L.
- Training takes 0.2K GPU hours on 8 V100 GPUs versus 120K GPU hours for GLIP-L, with 35M learnable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate SAM position-aware tokens and CLIP semantics in RegionSpot | LVIS val with GT boxes, compared with CLIP-L w/ mask · LVIS val, GT boxes | AP_all: 49.2 -> 59.9 | Helped |
| Use RegionSpot-ProPlus instead of GLIP-L | LVIS val with GLIP-L proposals · LVIS val, GLIP-L proposals | AP_r: +13.1 AP | Helped |
| Use RegionSpot-ProPlus instead of GroundingDINO-L | LVIS minival · LVIS minival | AP_r: +11.0 | Helped |
| Use RegionSpot-ProPlus instead of GLIP-L | ODinW with GLIP-L proposals · ODinW average | AP: +2.5 AP | Helped |
| Add CLIP feature map instead of removing CLIP vision features (ablation) | RegionSpot-BL · LVIS ablation, GLIP boxes | AP: 8.0 -> 22.1 | Helped |
| Add the CLIP class token to the CLIP feature map (ablation) | RegionSpot-BL · LVIS ablation, GLIP boxes | AP: 22.1 -> 23.7 | Helped |
| Extract position-aware tokens after the SAM transformer rather than the prompt encoder (ablation) | RegionSpot-BL · LVIS ablation, GLIP boxes | AP: 18.6 -> 23.7 | Helped |
| Extract position-aware tokens after the SAM transformer rather than the MLP (ablation) | RegionSpot-BL · LVIS ablation, GLIP boxes | AP: 20.4 -> 23.7 | Helped |
| Use 3 blocks rather than 1 block (ablation) | RegionSpot-BL · LVIS ablation, GLIP boxes | AP: 23.2 -> 23.7 | Helped |
| Use 6 blocks rather than 3 blocks (ablation) | RegionSpot-BL · LVIS ablation, GLIP boxes | AP: 23.7 -> 22.8 | Hurt |
| Add multiple box prompts to SAM (ablation) | RegionSpot-BL · LVIS prompt ablation | AP: 18.5 -> 22.1 | Helped |
| Add text prompts (ablation) | RegionSpot-BL with multiple box prompts · LVIS prompt ablation | AP: +1.6 AP | Helped |
Surprising findings
- Using SAM proposals instead of GT boxes lowers the RegionSpot-Pro+ AP_all from 59.9 to 18.2, showing strong dependence on proposal quality.
Limitations
- Relies on external region proposals (GLIP or GT boxes) rather than SAM's own automatic localization.
- Performance drops sharply with noisy SAM grid proposals (AP_all 18.2 vs 59.9 with GT boxes).
- Mask AP results are evaluated in a limited zero-shot setting using GLIP boxes.
Experimental setup
Datasets: Objects365 OpenImages V3Det LVIS ODinW COCO
Benchmarks: LVIS LVIS minival ODinW
Metrics: AP AP_r AP_f AP_all mAP
Baselines: CLIP-L w/ mask GLIP-T GLIP-L GroundingDINO-L RegionCLIP F-VLM Detic-ViLD ViLD X-Decoder OpenSeeD
Builds on: CLIP SAM GLIP RegionCLIP DETR
Closest papers by meaning
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.88)
- REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders (NeurIPS 2025 · similarity 0.88)
- RegionReasoner: Region-Grounded Multi-Round Visual Reasoning (ICLR 2026 · similarity 0.87)
- RegionGPT: Towards Region Understanding Vision Language Model (CVPR 2024 · similarity 0.87)
- Mechanisms of Object Localization in Vision-Language Models (CVPR 2026 · similarity 0.87)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.