Research map / Label-efficient and robust learning / Self-supervised and contrastive
SOHES: Self-supervised Open-world Hierarchical Entity Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOHES, a self-supervised open-world hierarchical entity segmentation method that clusters DINO ViT-B/8 features into pseudo-labels, trains a Mask2Former segmenter with an ancestor-prediction head, and refines it via teacher-student self-correction using raw SA-1B images.
Problem
Open-world entity segmentation normally relies on costly human-annotated masks (e.g., SAM trained on SA-1B), and self-supervised representations do not directly produce region-level, entity-and-part segmentation.
Approach
Three self-evolving phases: hierarchical feature clustering of DINO patches (global-to-local, with CascadePSP refinement) to create pseudo-labels with part-whole hierarchy; training Mask2Former with an added ancestor-relation head on these pseudo-labels; and teacher-student mutual learning with a area-dependent dynamic confidence threshold to correct noise.
Key results
- SOHES reaches 33.3 AR on SA-1B versus 26.0 for the prior self-supervised state of the art HASSOD, using only 2% of raw SA-1B images.
- On zero-shot whole-entity benchmarks SOHES reports 30.5 AR on COCO, 29.1 on LVIS, 31.1 on ADE20K and 33.5 on EntitySeg, improving over HASSOD by 2.2 to 7.3 AR.
- On PACO-LVIS part segmentation SOHES reaches 17.1 AR versus 11.4 for HASSOD and 8.9 for CutLER; on PartImageNet it reaches 36.0 AR, above SAM's 28.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SOHES instead of HASSOD | self-supervised open-world entity segmentation · SA-1B | AR: 26.0 -> 33.3 (+7.3) | Helped |
| use SOHES instead of HASSOD | self-supervised open-world entity segmentation · EntitySeg | AR: 26.2 -> 33.5 (+7.3) | Helped |
| use SOHES instead of HASSOD | self-supervised part segmentation · PACO-LVIS | AR: 11.4 -> 17.1 (+5.7) | Helped |
| use an area-dependent dynamic confidence threshold instead of a fixed threshold (ablation) | teacher-student self-correction · SA-1B | AR: 32.5 -> 33.3 | Helped |
| use an area-dependent dynamic confidence threshold instead of a fixed threshold (ablation) | teacher-student self-correction · SA-1B | AR_S: 6.3 -> 8.6 | Helped |
| mix global clustering outputs across thresholds and apply NMS rather than use only one threshold (ablation) | self-exploration pseudo-label generation · SA-1B initial pseudo-labels | AR: 8.1 -> 9.1 | Helped |
| add local re-clustering of small candidate regions (ablation) | self-exploration pseudo-label generation · SA-1B initial pseudo-labels | AR: 9.1 -> 11.6 | Helped |
| replace CascadePSP refinement with DenseCRF (ablation) | self-exploration pseudo-label generation · SA-1B initial pseudo-labels | AR: 16.4 -> 12.0 | Hurt |
| replace CascadePSP refinement with CRM (ablation) | self-exploration pseudo-label generation · SA-1B initial pseudo-labels | AR: 16.4 -> 14.1 | Hurt |
| use DINO ViT-B patch size 8 instead of patch size 32 (ablation) | global clustering for initial pseudo-labels · SA-1B initial pseudo-labels | AR: 3.3 -> 7.6 | Helped |
| remove the ancestor prediction head (ablation) | SOHES · SA-1B | AR: 33.3 -> 33.0 | No clear effect |
| replace Mask2Former with Cascade Mask R-CNN (ablation) | SOHES segmentation model · SA-1B | AR: 33.3 -> 31.9 | Hurt |
Surprising findings
- SOHES outperforms SAM in AR on PartImageNet (36.0 vs 28.3) and is near SAM on PACO, because it predicts more parts and subparts than SAM.
- Vanilla teacher-student self-correction with a fixed confidence threshold lowers overall AR (32.5 vs 32.6), requiring the area-dependent dynamic threshold to gain.
Limitations
- Fails to associate disconnected or occluded segments of the same entity, since only adjacent regions are merged during self-exploration.
- Produces imprecise masks for letters and characters and misses blurred backgrounds.
- Pseudo-label generation is costly: ViT-B/8 is the best option only because smaller patch sizes were too expensive to compute.
Experimental setup
Datasets: SA-1B
Benchmarks: MS-COCO LVIS ADE20K EntitySeg SA-1B PartImageNet PACO-LVIS
Metrics: average recall (AR) average precision (AP)
Baselines: SAM CutLER HASSOD FreeSOLO
Builds on: DINO Mask2Former ViT-Adapter CascadePSP CutLER
Closest papers by meaning
- EntitySAM: Segment Everything in Video (CVPR 2025 · similarity 0.86)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.86)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.86)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.85)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.85)
- MOFI: Learning Image Representations from Noisy Entity Annotated Images (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.