Research map / Detection and segmentation / Open-set and few-shot detection
Learning Hierarchical Image Segmentation For Recognition and By Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAST, a ViT variant using SEEDS superpixel tokens and graph pooling that learns hierarchical part-whole segmentation purely from image-level recognition objectives (self-supervised MoCo or supervised), matching or beating ViT, HSG, SAM on segmentation and classification with lower compute.
Problem
Vision models trained on image-level recognition do not produce consistent hierarchical part-whole segmentation, while segmentation models are trained separately with pixel labels and cannot be directly used for recognition.
Approach
Replace ViT square patches with superpixel segment tokens and insert graph pooling modules that progressively merge segments fine-to-coarse, so the segmentation hierarchy is an internal byproduct of the recognition model trained only with image-level objectives; test-time entropy minimization can further refine segmentation.
Key results
- On PartImageNet object segmentation, unsupervised CAST-B reaches 29.66 region mIoU vs. 21.97 for SAM-H and 25.34 for ViT-B, with 13 vs. 3166 GFLOPS for SAM-H.
- On ImageNet-1K linear probing, CAST-S achieves 68.1% top-1 at 3.4 GFLOPS vs. ViT-S 67.9% at 4.7 GFLOPS and Swin-T 63.0% at 4.5 GFLOPS.
- Unsupervised CAST-S on PASCAL VOC before fine-tuning reaches 38.4 region mIoU vs. 30.9 for ViT-S.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CAST's hierarchical segment tokens and graph pooling instead of ViT tokens | CAST-B vs. ViT-B · PartImageNet object segmentation | region mIoU: 25.34 -> 29.66 | Helped |
| use CAST instead of SAM-H | PartImageNet segmentation · PartImageNet object segmentation | region mIoU: 21.97 -> 29.66 | Helped |
| use CAST instead of HSG for hierarchical parsing | CAST-S · DensePose 1-label parsing | F-score: 26.2 -> 33.2 | Helped |
| replace ViT patches and fixed token counts with superpixels and hierarchical token pooling | CAST-S · PASCAL-VOC before tuning | region mIoU: 30.9 -> 38.4 | Helped |
| use CAST instead of ViT | ImageNet-100 linear probing · ImageNet-100 | top-1 accuracy: 78.1 -> 79.9 | Helped |
| use CAST instead of ViT | ImageNet-100 classification · ImageNet-100 | GFLOPS: 4.7 -> 3.4 | Helped |
| replace Graph Pooling with Random Sampling (ablation) | CAST-S · ImageNet-100 linear probing | accuracy: 79.9 -> 55.8 | Hurt |
| replace Graph Pooling with K-Means (ablation) | CAST-S · ImageNet-100 linear probing | accuracy: 79.9 -> 73.9 | Hurt |
| replace Graph Pooling with ToMe (ablation) | CAST-S · ImageNet-100 linear probing | accuracy: 79.9 -> 78.1 | Hurt |
| use Farthest Point Sampling rather than Random Sampling for centroid initialization (ablation) | CAST-S · ImageNet-100 linear probing | accuracy: 71.2 -> 79.9 | Helped |
| use token granularities 196, 64, 32, 16 rather than 196, 32, 16, 8 (ablation) | CAST-S · ImageNet-100 linear probing | accuracy: 78.8 -> 79.9 | Helped |
| use SEEDS rather than SLIC superpixels (ablation) | CAST · ImageNet-1K classification | top-1 accuracy: 65.6 -> 68.1 | Helped |
Surprising findings
- CAST underperforms ViT-S on Cityscapes (72.1% vs 74.6%), attributed to off-the-shelf superpixels missing thin structures.
- SAM's boundary F-score is higher than CAST's, though CAST wins on region mIoU for objects and parts.
Limitations
- Relies on fixed off-the-shelf superpixels (SEEDS), which miss thin structures such as poles in street scenes.
- Underperforms ViT-S on Cityscapes semantic segmentation.
- Graph pooling's farthest point sampling adds latency overhead, and SEEDS superpixel generation is not GPU-optimized.
Experimental setup
Datasets: ImageNet-1K ImageNet-100 COCO PartImageNet DensePose PASCAL VOC ADE20K PASCAL-Context
Benchmarks: PartImageNet DensePose PASCAL VOC ADE20K PASCAL-Context ImageNet-1K ImageNet-100
Metrics: region mIoU boundary F-score precision recall F-score top-1 linear probing accuracy GFLOPS
Baselines: ViT SAM HSG Swin Transformer GroupViT ToMe FINCH
Builds on: Vision Transformer MoCo DeiT SEEDS superpixels Segmenter
Closest papers by meaning
- Deep Hierarchical Semantic Segmentation (CVPR 2022 · similarity 0.91)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.91)
- Self-Supervised Learning of Object Parts for Semantic Segmentation (CVPR 2022 · similarity 0.90)
- Hierarchical Prototype Learning for Semantic Segmentation (ICLR 2026 · similarity 0.90)
- Flattening the Parent Bias: Hierarchical Semantic Segmentation in the Poincare Ball (CVPR 2024 · similarity 0.89)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.