Research map / Detection and segmentation / Open-set and few-shot detection

Prompting Diffusion Models for Zero-Shot Instance Segmentation

NeurIPS 2026 · Detection and segmentation › Open-set and few-shot detection · approach: Prompted SAM segmentation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Prompt2Seg adds a ControlNet branch to a frozen Gen2Seg diffusion segmenter so spatial point prompts (Gaussian maps) condition the output directly, yielding zero-shot interactive instance segmentation trained only on synthetic Hypersim and Virtual KITTI 2 data.

Problem

Promptable (interactive) segmentation with discriminative models over-segments objects with strong internal edges or no clear boundaries, while existing generative segmenters (Gen2Seg) use user clicks only in post-hoc color-based retrieval rather than conditioning the generation on user intent.

Approach

Inject user spatial prompts (two-scale Gaussian point maps or dense confidence maps) into a frozen diffusion segmentation U-Net via a trainable ControlNet branch, so the model predicts only the prompted instance before decoding instead of coloring all instances and selecting one afterward.

Key results

• Zero-shot mIoU on DRAM 53.1 and PIDRay 47.5, exceeding SAM (51.6 and 44.2), while Gen2Seg scores 47.5 and 34.9.
• On COCO large objects, P2Seg reaches 62.3 mIoU versus SAM 57.6; on COCO small objects it scores 12.1 versus SAM 57.0.
• On RoadAnomaly, Mask2Former + P2Seg raises AP from 78.45 to 88.41 and lowers FPR from 11.83 to 4.95.

Surprising findings

• Prompt2Seg outperforms SAM ViT-H on DRAM, PIDRay and ZeroWaste despite training only on synthetic data, while SAM (trained on SA-1B) remains better on COCO medium and small objects.
• Conditioning the diffusion model on dense uncertainty maps refines noisy masks, improving Mask2Former AP on RoadAnomaly.

Limitations

• Struggles with small objects due to latent diffusion resolution bottleneck (COCO small mIoU 12.1 vs SAM 57.0).
• Single spatial prompt cannot always convey desired granularity; multi-prompt and negative prompts are future work.
• Still lags behind SAM on several in-distribution-like benchmarks (COCO, EgoHOS, VOC, HRSOD).

Experimental setup

Datasets: Hypersim Virtual KITTI 2

Benchmarks: COCO DRAM EgoHOS PIDRay Pascal VOC HRSOD ZeroWaste RoadAnomaly

Metrics: mIoU AP FPR Boundary F Boundary IoU

Baselines: SAM SimpleClick Gen2Seg MAE-H P2Seg-Concat P2Seg-Adapter

Builds on: Gen2Seg ControlNet Stable Diffusion SimpleClick

Closest papers by meaning

• Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation (CVPR 2025 · similarity 0.90)
• Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.89)
• Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.88)
• Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.88)
• Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.88)
• AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.88)