Research map / Detection and segmentation / Open-set and few-shot detection
Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds monocular depth guidance to STEGO-style unsupervised semantic segmentation via a depth-feature correlation loss and farthest-point sampling on depth point clouds (DepthG).
Problem
Unsupervised semantic segmentation methods distill features via feature correlation but ignore the 3D spatial layout of scenes, and random feature sampling is inefficient.
Approach
Extends STEGO's feature correlation loss with a depth correlation tensor computed from zero-shot monocular depth maps (ZoeDepth), so feature similarity is pushed by 3D distance; uses farthest-point sampling on the depth point cloud to select features, and optionally a depth-based local hidden positives propagation (3D-LHP). Depth is used only during training.
Key results
- On COCO-Stuff 27 with ViT-S/8, Ours w/ 3D-LHP reaches 26.7 unsupervised mIoU vs 24.5 for STEGO, and Ours reaches 56.3 unsupervised accuracy vs 48.3.
- On COCO-Stuff with ViT-B/8, Ours improves STEGO unsupervised mIoU from 28.2 to 29.0.
- On Cityscapes, Ours reaches 23.1 unsupervised mIoU vs 21.0 for STEGO (ViT-B/8), and 81.6 unsupervised accuracy vs 73.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Depth-Feature Correlation and FPS | STEGO with ViT-S/8 · COCO-Stuff 27 | unsupervised accuracy: 48.3 -> 56.3 | Helped |
| Add Depth-Feature Correlation and FPS | STEGO with ViT-S/8 · COCO-Stuff 27 | unsupervised mIoU: 24.5 -> 25.6 | Helped |
| Add Depth-Feature Correlation and FPS | STEGO with ViT-B/8 · COCO-Stuff 27 | unsupervised mIoU: 28.2 -> 29.0 | Helped |
| Add Depth-Feature Correlation and depth-guided sampling | STEGO with ViT-B/8 · Cityscapes | unsupervised mIoU: 21.0 -> 23.1 | Helped |
| Add Depth-Feature Correlation and FPS | STEGO with ViT-S/8 · Potsdam-3 | unsupervised accuracy: 77.0 -> 80.4 | Helped |
| Add 3D-LHP to Depth-Feature Correlation + FPS (ablation) | DepthG with ViT-S/8 · COCO-Stuff 27 | unsupervised mIoU: 25.6 -> 26.7 | Helped |
| Add 3D-LHP to Depth-Feature Correlation + FPS (ablation) | DepthG with ViT-S/8 · COCO-Stuff 27 | unsupervised accuracy: 56.3 -> 55.1 | Hurt |
| Use attention-based LHP instead of no LHP (ablation) | DepthG with ViT-S/8 · COCO-Stuff 27 | unsupervised mIoU: 25.6 -> 24.5 | Hurt |
| Use attention-based LHP instead of no LHP (ablation) | DepthG with ViT-S/8 · COCO-Stuff 27 | unsupervised accuracy: 56.3 -> 52.6 | Hurt |
| Use MiDaS instead of ZoeDepth to generate depth maps (ablation) | DepthG with ViT-S/8 · COCO-Stuff 27 | unsupervised mIoU: 25.6 -> 25.0 | Hurt |
| Use KBR instead of ZoeDepth to generate depth maps (ablation) | DepthG with ViT-S/8 · COCO-Stuff 27 | unsupervised mIoU: 25.6 -> 23.1 | Hurt |
Surprising findings
- Using attention-based local hidden positives instead of depth-based propagation lowers performance (24.5 mIoU), suggesting depth-consistent propagation matters.
- Depth-feature correlation with FPS only modestly changes unsupervised mIoU on COCO-Stuff (24.5 to 25.6), while accuracy gains are larger.
Limitations
- Not applicable to domains where monocular depth estimation is unsuitable, such as CT scans and medical data.
- Aerial top-down imagery (Potsdam-3) is out-of-distribution for the depth estimator, limiting gains there.
- Gains over STEGO on COCO-Stuff are modest in unsupervised mIoU for ViT-B/8 (+0.8).
Experimental setup
Datasets: COCO-Stuff Cityscapes Potsdam-3 ImageNet-1k
Benchmarks: COCO-Stuff 27 Cityscapes Potsdam-3
Metrics: unsupervised accuracy unsupervised mIoU linear probing accuracy linear probing mIoU
Baselines: STEGO STEGO + HP IIC PiCIE DINO DINOSAUR TransFGU ACSeg
Builds on: STEGO Hidden Positives DINO ZoeDepth
Closest papers by meaning
- Towards Scene Understanding: Unsupervised Monocular Depth Estimation With Semantic-Aware Representation (CVPR 2019 · similarity 0.90)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.89)
- Integrating Low-Level Visual Cues for Enhanced Unsupervised Semantic Segmentation (AAAI 2025 · similarity 0.88)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.87)
- LogoSP: Local-global Grouping of Superpoints for Unsupervised Semantic Segmentation of 3D Point Clouds (CVPR 2025 · similarity 0.87)
- Unsupervised Universal Image Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.