Research map / Detection and segmentation / Open-set and few-shot detection
Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a hierarchy-agnostic unsupervised segmentation method that recursively partitions a self-supervised feature graph via spectral clustering with perturbation-based stopping, producing a region tree, and introduces NMCovering and NHCovering metrics.
Problem
Unsupervised semantic segmentation is ambiguous because semantic granularity varies, and existing methods depend on dataset-specific priors or fixed numbers of clusters.
Approach
Recursive spectral partitioning of a graph built from self-supervised patch features, with the number of subgraph components chosen by eigengap and a matrix-perturbation (Davis-Kahan / Yu et al.) stability criterion, yielding an unconstrained semantic region tree; plus two new covering metrics that evaluate across hierarchy levels.
Key results
- On PascalVOC2012 val, the method (with CRF, ViT-B14 backbone) reaches 80.3 ± 1.1 mIoU, versus 41.7 for Leopart and 41.9 for HSG among unsupervised baselines.
- Granularity-agnostic NMCovering on PascalVOC2012 is 75.4 and on Potsdam 56.3; on COCO-Stuff hierarchy-agnostic NHCovering is 42.9.
- Replacing semantic-smoothness hierarchy with boundary-potential methods (SE-OWT-UCM, PMI-OWT-UCM) lowers PascalVOC2012 NMCovering to 59.0 and 61.3 versus 75.4 for the proposed method without CRF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CRF boundary sharpening (ablation) | Ours · PascalVOC2012 | mIoU: 78.1 -> 80.3 | Helped |
| add CRF boundary sharpening (ablation) | Ours · COCO-Stuff | mIoU: 58.7 -> 59.9 | Helped |
| use the DINOv2-ViT-B14-REG backbone instead of other tested backbones (ablation) | Ours · PascalVOC2012 | mIoU: 76.2 -> 78.1 | Helped |
| remove superpixel overclustering (ablation) | Ours · PartImageNet | NMCovering: 63.1 -> 65.7 | Helped |
| increase spectral overclustering size from 50 to 300 (ablation) | Ours · PascalVOC2012 | mIoU: 76.3 -> 78.1 | Helped |
| increase spectral overclustering size from 50 to 300 (ablation) | Ours · Cityscapes | mIoU: 30.3 -> 48.8 | Helped |
| set kmin to 5 instead of 1 (ablation) | Ours · PartImageNet | NMCovering: 63.1 -> 59.9 | Hurt |
| use λmax=.8 instead of .5 (ablation) | Ours · COCO-Stuff | NHCovering: 36.9 -> 41.1 | Helped |
| increase pmax from 15 to 20 (ablation) | Ours · COCO-Stuff | NHCovering: 41.1 -> 42.9 | Helped |
| use the proposed recursive spectral method instead of boundary-potential methods | Ours · PascalVOC2012 | NMCovering: 61.3 -> 75.4 | Helped |
Surprising findings
- Low-level boundary-based hierarchical grouping (SE/PMI OWT-UCM) performs much worse than semantic-feature grouping, suggesting low-level cues are inappropriate for semantic hierarchies.
- Pascal-Part hierarchy-agnostic NMCovering is only 39.5, attributed to coarser part annotations relative to PartImageNet.
Limitations
- Segmentation quality and runtime scale with input size
- Small object parts are hard to detect, especially with over-clustering
- Susceptible to biases in self-supervised pretraining data
Experimental setup
Datasets: PascalVOC2012 MSCOCO COCO-Stuff Cityscapes KITTI-STEP Mapillary Vistas Potsdam Vaihingen Pascal-Part PartImageNet PartImageNet-158 KITTI-SS
Benchmarks: PascalVOC2012 COCO-Stuff Cityscapes KITTI-STEP Mapillary Vistas Potsdam Pascal-Part PartImageNet PartImageNet-158 MSCOCO
Metrics: mIoU pAcc mAcc fIoU NMCovering NHCovering NFCovering
Baselines: STEGO SmooSeg HSG Leopart CUTLER SE-OWT-UCM PMI-OWT-UCM MaskDistill
Builds on: normalized cut Ng-Jordan-Weiss spectral clustering Leopart HSG DINOv2
Closest papers by meaning
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.91)
- Learning Hierarchical Image Segmentation For Recognition and By Recognition (ICLR 2024 · similarity 0.91)
- Flattening the Parent Bias: Hierarchical Semantic Segmentation in the Poincare Ball (CVPR 2024 · similarity 0.90)
- Integrating Low-Level Visual Cues for Enhanced Unsupervised Semantic Segmentation (AAAI 2025 · similarity 0.90)
- Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling (CVPR 2024 · similarity 0.89)
- unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.