Research map / Detection and segmentation / Open-set and few-shot detection

Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation

CVPR 2025 · Detection and segmentation › Open-set and few-shot detection · approach: Interactive and referring segmentation · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes M2N2, a training-free unsupervised interactive point-prompt segmentation method that interprets Stable Diffusion 2 self-attention as a Markov transition operator to build Markov-maps combined with a truncated nearest-neighbor rule.

Problem

Interactive point-prompt image segmentation without any segmentation labels or training, reducing the number of clicks (NoC) needed for high-quality masks; prior unsupervised methods rely on self-supervised backbones and pseudo-label training.

Approach

Treat the aggregated self-attention tensor of Stable Diffusion 2 as a doubly stochastic Markov transition matrix; for each prompt point, count per-pixel iterations needed to reach a relative probability threshold (Markov-map), refine with a modified flood fill for instance awareness, and segment via a truncated 1-NN with adaptively chosen thresholds.

Key results

• M2N2 achieves NoC85 of 1.62 on GrabCut and 3.88 on Berkeley, with NoC90 of 1.90 and 7.72... (see table: GrabCut NoC85 1.62, Berkeley 2.45/3.88 range across datasets)
• On DAVIS, M2N2 reduces NoC85 to 4.60 and NoC90 to 6.72, beating the previous best unsupervised trained method MIS (6.33 / 8.44) by 1.73 and 1.72 clicks.
• Despite being training-free, M2N2 outperforms unsupervised training-based MIS on three of four datasets.

Measured effects reported in the paper

Surprising findings

• Higher attention resolution (128x128 via 1024x1024 input) improves NoC even though it exceeds SD2's 768x768 training resolution.
• Shows a significant domain bias on medical data (BraTS, OAIZIB) compared with SimpleClick.

Limitations

• Low attention resolution makes thin and fine structures hard to segment, requiring many more clicks.
• Flood fill splits instances under obstruction and fails for overlapping semantically equal instances; requires non-overlapping instances per Markov-map.
• Significant domain bias toward natural images, weak on medical data.

Experimental setup

Benchmarks: GrabCut Berkeley SBD DAVIS

Metrics: NoC85 NoC90

Baselines: MIS TokenCut FreeMask DSM SimpleClick CPlot GraphCut Random Walk Geodesic Matting GSC ESC Attention-NN

Builds on: DiffSeg MIS KL-divergence attention matching Just-in-time bilateral upsampling (JBU) SimpleClick

Closest papers by meaning

• Prompting Diffusion Models for Zero-Shot Instance Segmentation (NeurIPS 2026 · similarity 0.90)
• Diffuse Attend and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion (CVPR 2024 · similarity 0.89)
• Attention (as Discrete-Time Markov) Chains (NeurIPS 2025 · similarity 0.88)
• Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.87)
• NTClick: Achieving Precise Interactive Segmentation With Noise-tolerant Clicks (CVPR 2025 · similarity 0.87)
• Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.87)