Research map / Detection and segmentation / Open-set and few-shot detection
Boosting Segment Anything Model Towards Open-Vocabulary Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sambor integrates SAM's image encoder with a two-stage open-vocabulary detector via a CLIP-informed SideFormer adapter and an Open-set RPN that adds SAM-derived proposals, evaluated zero-shot on COCO and LVIS.
Problem
SAM is class-agnostic and cannot recognize object semantics, while open-vocabulary detectors rely on box annotations for localization generalization, so neither gives both localization and classification zero-shot ability in one end-to-end model.
Approach
Build an end-to-end open-vocabulary detector on the frozen SAM ViT-B encoder, add a ladder side transformer (SideFormer) that extracts object-aware SAM features and injects CLIP RN50x64 visual semantics, and augment RPN proposals with SAM automatic-mask-generation boxes (Open-set RPN) so localization generalizes too.
Key results
- Sambor trained on O365 reaches 47.3 box AP on COCO val zero-shot, +2.4 over GLIP-T (C) style comparison and +0.6 over G-DINO-T; Open-set RPN variant reaches 48.6 AP.
- On LVIS MiniVal with O365 only, Sambor Open-set RPN reaches 33.1 AP, 15.3 above GLIP-T and 4.3/4.5 above DetCLIP/DetCLIPv2.
- With O365+GoldG fine-tuning, Sambor reaches 39.6 AP on LVIS MiniVal and 32.8 on Val v1.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the SAM Extractor (ablation) | Sambor baseline · COCO val | AP: 39.0 -> 42.4 | Helped |
| Add the SAM Extractor (ablation) | Sambor baseline · LVIS MiniVal | AP: 27.7 -> 29.0 | Helped |
| Add the CLIP Injector to SAM Extractor (ablation) | Sambor · COCO val | AP: 42.4 -> 47.3 | Helped |
| Add the CLIP Injector to SAM Extractor (ablation) | Sambor · LVIS MiniVal | AP: 29.0 -> 32.7 | Helped |
| Add SAM proposals without fine-tuning the detector on them (ablation) | Sambor with vanilla RPN · COCO val | AR@1000: 65.4 -> 67.5 (+2.1) | Helped |
| Add SAM proposals without fine-tuning the detector on them (ablation) | Sambor with vanilla RPN · COCO val | AP: 47.3 -> 46.7 (-0.6) | Hurt |
| Add SAM proposals without fine-tuning the detector on them (ablation) | Sambor with vanilla RPN · LVIS MiniVal | AR@1000: 49.1 -> 54.8 (+5.7) | Helped |
| Add SAM proposals without fine-tuning the detector on them (ablation) | Sambor with vanilla RPN · LVIS MiniVal | AP: 32.7 -> 29.3 (-3.4) | Hurt |
| Add SAM proposals and fine-tune on them (ablation) | Sambor with vanilla RPN · COCO val | AP: 47.3 -> 48.6 (+1.3) | Helped |
| Add SAM proposals and fine-tune on them (ablation) | Sambor with vanilla RPN · LVIS MiniVal | AP: 32.7 -> 33.1 (+0.4) | Helped |
| Use proposals only from SAM instead of combining SAM and RPN proposals (ablation) | Sambor with Open-set RPN · COCO 2017val | AP: 48.6 -> 46.8 | Hurt |
| Fine-tune the O365 model on GoldG | Sambor · LVIS MiniVal | AP: 33.1 -> 39.6 | Helped |
Surprising findings
- Adding Open-set RPN proposals without fine-tuning the detection head raised proposal AR@1000 on COCO (+2.1) and LVIS (+5.7) but lowered AP (COCO -0.6, LVIS -3.4).
- SAM-only proposals underperform on small objects (COCO AP_s 29.8 vs 33.5 for RPN), so a trainable RPN is needed.
Limitations
- Open-set RPN without fine-tuning hurts detection AP, requiring an extra fine-tuning stage.
- SAM-only proposals miss small objects; denser point grids are computationally costly.
- Zero-shot evaluation on COCO relies on O365 pre-training that covers all 80 COCO categories.
Experimental setup
Datasets: Objects365 GoldG Bamboo COCO LVIS
Benchmarks: COCO 2017 val LVIS MiniVal LVIS Val v1.0
Metrics: AP AP50 AP75 AR@1000 APr/APc/APf
Baselines: GLIP-T Grounding DINO-T DetCLIP-T DetCLIPv2-T X-Decoder OpenSeeD
Builds on: Segment Anything Model GLIP ViTDet Cascade R-CNN CLIP
Closest papers by meaning
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.90)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.90)
- Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts (NeurIPS 2024 · similarity 0.89)
- Grounding Everything: Emerging Localization Properties in Vision-Language Transformers (CVPR 2024 · similarity 0.89)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.89)
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.