Research map / Detection and segmentation / Weakly supervised detection and segmentation
Segment Anything without Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UnSAM, an unsupervised promptable and whole-image segmentation model trained on divide-and-conquer pseudo masks (CutLER top-down plus iterative bottom-up merging), and UnSAM+, which merges these masks with SA-1B ground truth to beat supervised SAM.
Problem
Segment Anything Model requires labor-intensive manual mask annotation (SA-1B), limiting scaling and causing annotator bias that misses small entities and parts; the question is whether comparable multi-granular segmentation can be learned without supervision.
Approach
A divide-and-conquer pseudo-mask pipeline: top-down NCuts-based CutLER gives instance/semantic masks, then bottom-up iterative merging of DINO ViT-B/8 patch features within each mask yields a multi-granular hierarchy used as training labels for Mask2Former (whole-image) and Semantic-SAM (promptable).
Key results
- UnSAM with ResNet-50 trained on 1% of SA-1B averages 39.2 to 41.1 AR across whole-image datasets, about 11.0 AR above the prior unsupervised SOTA.
- UnSAM+ (pseudo masks merged with SA-1B ground truth) exceeds SAM's AR by 6.7% and AP by 3.9% on SA-1B.
- On COCO promptable segmentation, UnSAM+ (Swin-Tiny) reaches 52.4 MaxIoU vs. SAM's 52.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UnSAM's divide-and-conquer pseudo-mask training instead of SOHES | UnSAM · Whole-image benchmarks (average) | AR: 30.1 -> 41.1 | Helped |
| Increase UnSAM training data from 0.1M to 0.4M images | UnSAM · Whole-image benchmarks (average) | AR: 39.2 -> 41.1 | Helped |
| Add unsupervised pseudo masks to SA-1B ground-truth training data | UnSAM · Whole-image benchmarks (average) | AR: 42.1 -> 48.8 | Helped |
| Add unsupervised pseudo masks to SA-1B ground-truth training data | UnSAM · Whole-image benchmarks | AR: +2.6 COCO; +4.7 LVIS; -0.5 ADE; +3.9 Entity; +4.0 SA-1B; +17.7 PtIn; +14.2 PACO | Mixed |
| Add SA-1B ground-truth labels to pseudo-mask training | UnSAM · COCO Val2017 | MaxIoU: 40.3 -> 52.4 | Helped |
| Add SA-1B ground-truth labels to pseudo-mask training | UnSAM · COCO Val2017 | OracleIoU: 59.5 -> 69.5 | Helped |
| Use CascadePSP rather than CRF for pseudo-mask refinement (ablation) | UnSAM pseudo masks · SA-1B | AR1000: 15.3 -> 23.9 | Helped |
| Use CascadePSP rather than CRF for pseudo-mask refinement (ablation) | UnSAM pseudo masks · SA-1B | AR small: 2.3 -> 7.9 | Helped |
| Compare UnSAM with SAM | UnSAM · PartImageNet | AR: 28.3 -> 52.7 | Helped |
| Compare UnSAM with SAM | UnSAM · PACO | AR: 18.1 -> 29.7 | Helped |
Surprising findings
- Training on only 1% of SA-1B images with pseudo masks plus ground truth lets the model outperform fully supervised SAM on several benchmarks, including AR on small entities by 16.2%.
- Unsupervised UnSAM trained on 1% of SA-1B is within about 1% of supervised SAM's AR on whole-image segmentation across datasets.
Limitations
- Misses repetitive instances with similar texture in dense fine-grained scenes.
- Over-segments images, e.g. treating folds and shadows on clothing as entity boundaries.
- Iterative merging sometimes fails to associate disconnected parts of the same entity.
Experimental setup
Datasets: SA-1B COCO
Benchmarks: COCO LVIS ADE20K EntitySeg SA-1B PartImageNet PACO
Metrics: average recall average precision MaxIoU OracleIoU
Baselines: SAM CutLER SOHES FreeSOLO U2Seg Semantic-SAM
Builds on: CutLER MaskCut Mask2Former Semantic-SAM SAM
Closest papers by meaning
- Unsupervised Universal Image Segmentation (CVPR 2024 · similarity 0.91)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.91)
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.89)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.89)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.89)
- unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.