Research map / Detection and segmentation / Open-set and few-shot detection

DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut

NeurIPS 2024 · Detection and segmentation › Open-set and few-shot detection · approach: Few-shot segmentation · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiffCut, a training-free zero-shot unsupervised segmentation method that applies recursive Normalized Cut to features from the final self-attention block of the SSD-1B diffusion UNet encoder, with concept-based high-resolution upsampling.

Problem

Unsupervised zero-shot semantic segmentation without segmentation labels, target-dataset adaptation, or a fixed number of segments; prior graph-based object discovery methods need a predefined number of objects and prior diffusion-based methods are either attention-based (costly, prompt-dependent) o

Approach

Use diffusion UNet encoder features (final self-attention block) to build a cosine-similarity affinity with soft power thresholding, and recursively apply Normalized Cut with a cost threshold tau to adaptively determine the number of segments; then assign concepts to bilinearly upsampled features for pixel-level masks.

Key results

• DiffCut achieves mIoU of 65.2 on Pascal VOC, 56.5 on Pascal Context, 34.1 on COCO-Object, 49.1 on COCO-Stuff-27, 30.6 on Cityscapes and 44.3 on ADE20K, averaging +7.3 mIoU over the second-best baseline.
• DiffCut gains +5.5 mIoU over DiffSeg on COCO-Stuff and +9.4 mIoU on Cityscapes.
• SSD-1B UNet encoder reaches patch-level alignment AUC 0.83, above DINOv2 and CLIP candidates.

Measured effects reported in the paper

Surprising findings

• Diffusion UNet encoder features cluster better than DINOv2 and CLIP at patch level, even though the diffusion model is not trained on dense labels.
• Smaller SSD-Vega (240M encoder params) performs comparably to SSD-1B on most benchmarks.

Limitations

• Fixed hyperparameters are not optimal for every dataset due to varying annotation granularity; best results need tau tuning on annotated data.
• Diffusion backbone not trained for specialized domains such as biomedical imaging, may struggle with out-of-distribution images.
• Fully supervised end-to-end methods still offer better efficiency and accuracy.

Experimental setup

Datasets: Pascal VOC Pascal Context COCO-Object COCO-Stuff-27 Cityscapes ADE20K

Benchmarks: Pascal VOC Pascal Context COCO-Object COCO-Stuff-27 Cityscapes ADE20K

Metrics: mIoU

Baselines: MaskCut DiffSeg ReCo MaskCLIP STEGO ACSeg EAGLE U2Seg CLIP-DIY FreeSeg-Diff

Builds on: Normalized Cut DiffSeg MaskCut TokenCut PAMR

Closest papers by meaning

• High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity (ICLR 2025 · similarity 0.89)
• Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.89)
• Diffuse Attend and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion (CVPR 2024 · similarity 0.89)
• Unlocking the Potential of Pre-trained Vision Transformers for Few-Shot Semantic Segmentation through Relationship Descriptors (CVPR 2024 · similarity 0.88)
• Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.88)
• Spectral Prompt Tuning: Unveiling Unseen Classes for Zero-Shot Semantic Segmentation (AAAI 2024 · similarity 0.88)