Research map / Detection and segmentation / Open-set and few-shot detection
USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes USE, a data pipeline that curates multi-granularity segment-text pairs from COCO and Visual Genome with MLLM captions, grounding DINO and SAM, plus a lightweight segment embedding head on frozen CLIP and DINOv2 for open-vocabulary segmentation.
Problem
Open-vocabulary image segmentation needs accurate classification of class-agnostic segments (e.g., from SAM) at varied granularities into text-defined categories, but existing methods have limited segment-text supervision that mostly covers nouns and objects.
Approach
Automatically generate large-scale segment-text pairs at object and part granularity using an MLLM with a part-aware prompt, referring-expression grounding over whole captions, and SAM mask generation; then train a lightweight segment embedding head (mask pooling plus linear projection over frozen CLIP and DINOv2 features) with a segment-text contrastive loss.
Key results
- USE+SAM (COCO,VG) reaches 37.1 mIoU on ADE-150 versus 29.6 for OVSeg, and 30.9 average across four semantic segmentation benchmarks, the best average among compared methods.
- On PartImageNet open-vocabulary part segmentation, USE (trained on COCO) gets 6.2 mAP overall, beating VLPart trained on Pascal Part (4.5) and Pascal Part + ImageNet (5.4).
- Combining CLIP and DINOv2 in the image encoder improves ADE-150 mIoU to 32.6 from 30.2 (CLIP only) and 31.9 (DINOv2 only).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use USE+SAM instead of SAN | Open-vocabulary semantic segmentation · ADE-150 | mIoU: 32.1 -> 37.0 | Helped |
| Use USE+SAM instead of SAN | Open-vocabulary semantic segmentation · ADE-847 | mIoU: 12.4 -> 13.3 | Helped |
| Use USE+SAM instead of SAN | Open-vocabulary semantic segmentation · PC-59 | mIoU: 57.7 -> 57.8 | Helped |
| Use USE+SAM instead of SAN | Open-vocabulary semantic segmentation · PC-459 | mIoU: 15.7 -> 14.7 | Hurt |
| Add VG training data to COCO | USE+SAM · ADE-150 | mIoU: 37.0 -> 37.1 | Helped |
| Add VG training data to COCO | USE+SAM · ADE-847 | mIoU: 13.3 -> 13.4 | Helped |
| Add VG training data to COCO | USE+SAM · PC-59 | mIoU: 57.8 -> 58.0 | Helped |
| Add VG training data to COCO | USE+SAM · PC-459 | mIoU: 14.7 -> 15.0 | Helped |
| Combine CLIP and DINOv2 instead of using CLIP only (ablation) | USE image encoder · ADE-150 | mIoU: 30.2 -> 32.6 | Helped |
| Combine CLIP and DINOv2 instead of using CLIP only (ablation) | USE image encoder · ADE-847 | mIoU: 10.3 -> 11.3 | Helped |
| Combine CLIP and DINOv2 instead of using DINOv2 only (ablation) | USE image encoder · ADE-150 | mIoU: 31.9 -> 32.6 | Helped |
| Combine CLIP and DINOv2 instead of using DINOv2 only (ablation) | USE image encoder · ADE-847 | mIoU: 10.2 -> 11.3 | Helped |
Surprising findings
- OVSeg's performance declines when using SAM segments (26.6 to 24.9 average), suggesting SAM segments at varying granularities are harder to classify.
Limitations
- SAM is edge-oriented and struggles to separate parts with blurry boundaries (e.g., elephant head vs. neck), hurting head segmentation.
- Part segmentation is slightly worse than VLPart trained with human-annotated part data and ImageNet w/ Parts (6.2 vs 7.8 mAP).
Experimental setup
Datasets: COCO Visual Genome ADE20K Pascal Context PartImageNet
Benchmarks: ADE20K Pascal Context PartImageNet
Metrics: mIoU mAP
Baselines: OVSeg OVSeg+SAM SAN ODISE OpenSeg SimSeg LSeg+ ZegFormer VLPart
Builds on: COMM SAM OVSeg Kosmos-2 Grounding DINO
Closest papers by meaning
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.91)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.91)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.90)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.90)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.