Research map / Detection and segmentation / Open-set and few-shot detection
Diffuse Attend and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffSeg, a training-free, unsupervised, zero-shot segmentation method that iteratively merges self-attention maps from a pre-trained Stable Diffusion v1.4 U-Net using KL divergence and then applies non-maximum suppression.
Problem
Segmenting arbitrary images into masks without any annotations, unsupervised adaptation, language input, or auxiliary image pools (zero-shot unsupervised segmentation).
Approach
Self-attention tensors in a pre-trained diffusion U-Net encode object grouping (intra- and inter-attention similarity); DiffSeg aggregates attention across resolutions, iteratively merges anchor-point attention maps by symmetric KL divergence, and takes an argmax over proposals to produce masks, with no training and no clustering count to specify.
Key results
- On COCO-Stuff-27, DiffSeg (512) reaches 72.5% pixel accuracy and 43.6 mIoU, versus 46.1 ACC / 26.3 mIoU for prior zero-shot SOTA ReCo.
- On Cityscapes at 512 resolution, DiffSeg achieves 76.0 ACC and 21.2 mIoU, exceeding STEGO (73.2 / 21.0) and ReCo (74.6 / 19.3) in accuracy and mIoU for the 512 setting.
- DiffSeg outperforms K-Means baselines with oracle cluster counts (K-Means-S 62.6 ACC / 34.7 mIoU) on COCO-Stuff-27 without specifying the number of clusters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffSeg (512) instead of ReCo | zero-shot segmentation · COCO-Stuff-27 | pixel accuracy: 46.1 -> 72.5 | Helped |
| Use DiffSeg (512) instead of ReCo | zero-shot segmentation · COCO-Stuff-27 | mIoU: 26.3 -> 43.6 | Helped |
| Use DiffSeg (512) instead of K-Means-S | COCO-Stuff-27 · COCO-Stuff-27 | pixel accuracy: 62.6 -> 72.5 | Helped |
| Use DiffSeg (512) instead of K-Means-S | COCO-Stuff-27 · COCO-Stuff-27 | mIoU: 34.7 -> 43.6 | Helped |
| Use DiffSeg (512) instead of ReCo | zero-shot segmentation · Cityscapes | pixel accuracy: 74.6 -> 76.0 | Helped |
| Use DiffSeg (512) instead of STEGO | Cityscapes · Cityscapes | mIoU: 21.0 -> 21.2 | No clear effect |
| Aggregate attention maps proportionally across resolutions rather than using only 64×64 maps (ablation) | DiffSeg · segmentation examples | segmentation quality | Helped |
| Use only 64×64 attention maps instead of proportional aggregation (ablation) | DiffSeg · segmentation examples | segmentation consistency | Hurt |
| Use only 32×32 attention maps instead of proportional aggregation (ablation) | DiffSeg · segmentation examples | segmentation detail | Hurt |
| Use only 16×16 or 8×8 attention maps instead of proportional aggregation (ablation) | DiffSeg · segmentation examples | segmentation quality | Hurt |
Surprising findings
- Using only 8x8 or 16x16 attention maps yields no segmentation at all because the whole image merges into one object under the chosen hyper-parameters.
- DinoSeg, the same merging algorithm applied to DINO attention, performs worse than DiffSeg on COCO-Stuff-27 (39.1 vs 43.6 mIoU).
Limitations
- Performance on Cityscapes depends strongly on input resolution because of small classes such as light poles and traffic signs.
- Outputs are unlabeled masks; semantic labeling is only a proposed extension in the appendix.
- Hyper-parameters (aggregation weights, timestep, KL threshold) are set empirically.
Experimental setup
Datasets: COCO-Stuff-27 Cityscapes DomainNet SUN-RGBD
Benchmarks: COCO-Stuff-27 Cityscapes
Metrics: pixel accuracy mean IoU
Baselines: K-Means-C K-Means-S DBSCAN STEGO PiCLE ReCo MaskCLIP IIC
Builds on: Stable Diffusion DiffuMask STEGO ReCo
Closest papers by meaning
- Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation (CVPR 2025 · similarity 0.89)
- DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut (NeurIPS 2024 · similarity 0.89)
- Prompting Diffusion Models for Zero-Shot Instance Segmentation (NeurIPS 2026 · similarity 0.87)
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.87)
- Factorized Diffusion Architectures for Unsupervised Image Generation and Segmentation (NeurIPS 2024 · similarity 0.86)
- SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.