Research map / Detection and segmentation / Open-set and few-shot detection
Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively
Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, Chen Change Loy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| share one frozen CLIP backbone via SAM2CLIP distillation instead of separate SAM and CLIP backbones | Open-Vocabulary SAM · COCO open-vocab, GT box prompts | FLOPs / parameters: 3,545G -> 1,180G; 808M -> 304M | Improved |
| add CLIP2SAM on top of SAM2CLIP (ablation) | Open-Vocabulary SAM · COCO with Detic box prompts | AP: 28.7 -> 34.4 | Improved |
| use SAM2CLIP alone (shared backbone) without CLIP2SAM (ablation) | Open-Vocabulary SAM · COCO with Detic box prompts | AP: 29.5 -> 28.7 | Hurt |
| use light FPN with RoIAlign for CLIP2SAM instead of a classification token with cross attention (ablation) | CLIP2SAM · COCO open-vocab, GT box prompts | Recognition accuracy: 79.3 -> 84.3 | Improved |
| use multi-scale instead of single-scale features in the SAM2CLIP adapter (ablation) | SAM2CLIP · COCO open-vocab, GT box prompts | IoU: 73.6 -> 78.3 | Improved |
| use ViT-based CLIP as frozen backbone instead of convolutional CLIP (ablation) | Open-Vocabulary SAM · COCO open-vocab, GT box prompts | IoU: 78.1 -> 38.6 | Hurt |
| use larger CLIP backbone RN50x64 instead of RN50x16 (ablation) | Open-Vocabulary SAM · COCO open-vocab, GT box prompts | Recognition accuracy: 55.1 -> 54.1 | No clear effect |
| add CoOp-style adapters to combined SAM and CLIP baselines (ablation) | Feature-Crop baseline · COCO open-vocab, GT box prompts | Recognition accuracy: 55.1 -> 70.9 | Improved |
| scale training data to V3Det + LVIS + I-21k (ablation) | Open-Vocabulary SAM · Large-vocabulary scaling | Accuracy: 83.1 -> 83.3 | No clear effect |
| add Object365 to V3Det + LVIS + I-21k training (ablation) | Open-Vocabulary SAM · Large-vocabulary scaling | Accuracy: 83.3 -> 83.0 | No clear effect |
| Open-Vocabulary SAM vs SAM-Huge on LVIS mask quality with Detic | Open-Vocabulary SAM · LVIS mask quality with Detic | mask mAP: 43.5 -> 42.6 | Hurt |
Closest papers by meaning
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.91)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.91)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.90)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.89)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.88)
- Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
