Research map / Vision-language and multimodal / CLIP and zero-shot
OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends frozen SAM2 with a frozen BEiT-3 multi-modal encoder and a ~4.5M-parameter adapter (positional tie-breaker queries plus cross-attention) to perform open-vocabulary semantic, instance, panoptic and referring segmentation from language prompts.
Problem
Open-vocabulary segmentation requires grounding free-form language (category names or referring sentences) into masks, handling unseen categories and multiple instances per query, which SAM-style promptable models lack.
Approach
Feed frozen SAM2 mask decoder with language-conditioned queries from a frozen BEiT-3 early-fusion encoder; K learnable positional tie-breaker vectors added to the projected text embedding yield distinct queries for multi-instance output, refined by cross-attention to SAM2 level-3 features; also proposes an Oracle-Prompts evaluation protocol.
Key results
- Zero-shot ADE20K-857 mIoU 33.1 vs 9.2 for X-Decoder under global matching, and 4.0 points above X-Decoder under oracle prompts (29.1).
- RefCOCOg validation cIoU 74.0 with 4.5M trainable parameters, below EVF-SAM (77.0, 674M trainable) and UNINEXT (74.4).
- Frozen BEiT-3 with 4.5M adapter parameters (60.4 ADE-150 mIoU) outperforms fully fine-tuning BEiT-3 (23.6 mIoU) in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace late-fusion CLIP features with early-fusion BEiT-3 image-text encoding (ablation) | OpenWorldSAM · ADE-150 | mIoU: 26.5 -> 60.4 | Helped |
| Remove image input from BEiT-3 (text-only encoding) (ablation) | OpenWorldSAM · ADE-150 | mIoU: 60.4 -> 26.3 | Hurt |
| Fine-tune the BEiT-3 encoder rather than keep it frozen (ablation) | OpenWorldSAM · ADE-150 | mIoU: 60.4 -> 23.6 | Hurt |
| Add positional tie-breakers (ablation) | OpenWorldSAM · ADE-150 | AP: 1.0 -> 17.1 | Helped |
| Add cross-attention layers (ablation) | OpenWorldSAM · ADE-150 | mIoU: 56.8 -> 60.4 | Helped |
| Increase tie-breakers from K=10 to K=20 (ablation) | OpenWorldSAM · ADE-150 instance segmentation | AP: 14.2 -> 16.9 | Helped |
| Increase tie-breakers from K=20 to K=30 (ablation) | OpenWorldSAM · ADE-150 instance segmentation | AP: 16.9 -> 16.5 | Hurt |
| Increase tie-breakers from K=20 to K=30 (ablation) | OpenWorldSAM · ADE-150 instance segmentation | Average Recall@100: 28.8 -> 29.4 | Helped |
| Use 3 cross-attention layers instead of 1 (ablation) | OpenWorldSAM · ADE-150 | mIoU: 59.0 -> 60.4 | Helped |
| Add second-stage mask refinement (ablation) | OpenWorldSAM · ADE-150 | mIoU: 60.4 -> 58.0 | Hurt |
| Fine-tune OpenWorldSAM on Cityscapes | OpenWorldSAM · Cityscapes | mIoU: 39.4 -> 57.4 | Helped |
| Use OpenWorldSAM instead of X-Decoder (L) under oracle prompts | Open-vocabulary segmentation · ADE-857 | mIoU: 29.1 -> 33.1 | Helped |
Surprising findings
- Fine-tuning the full BEiT-3 encoder substantially degrades open-vocabulary performance (ADE-150 mIoU drops from 60.4 to 23.6), attributed to overfitting on sparse category-label prompts.
- Two-stage inference gives marginal metric gains despite visually refined mask contours.
Limitations
- Under-performs on outdoor driving datasets (Cityscapes 39.4 mIoU zero-shot vs 52.0 for X-Decoder) due to COCO-only training and resolution bottleneck.
- AP lags behind APE, which uses extra detection datasets such as Objects365.
- Number of instances per query capped by tie-breaker count K=20.
Experimental setup
Datasets: COCO2017-Stuff RefCOCOg
Benchmarks: ADE20K-150 ADE20K-857 PASCAL VOC-20 PASCAL Context-59 PASCAL Context-459 SUN-RGBD-37 ScanNet-20 ScanNet-40 RefCOCOg Cityscapes BDD10K
Metrics: PQ mAP mIoU cIoU
Baselines: X-Decoder APE OVSeg MaskCLIP OpenSeeD ESC-Net EVF-SAM LISA GLaMM u-LLaVA SEEM UNINEXT
Builds on: SAM2 BEiT-3 EVF-SAM X-Decoder DETR
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.91)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.91)
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.90)
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.90)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.90)
- Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.