Research map / Detection and segmentation / Open-set and few-shot detection
Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts Stable Diffusion 2.1 into a one-step image-to-mask generator for few-shot and in-context semantic segmentation via KV-fusion self-attention, support-mask concatenation, and white-on-black query mask supervision (DiffewS).
Problem
Few-shot semantic segmentation (FSS) and in-context segmentation need models that generalize to unseen categories while preserving fine-grained mask prediction; prior diffusion-based approaches add decoder heads or break the UNet priors.
Approach
Keep the Latent Diffusion UNet essentially unchanged: query image attends to support image via KV fusion in self-attention, support mask is concatenated as latent input, and the UNet directly regresses the query mask latent in one step (OI2M) with white-foreground/black-background supervision.
Key results
- In the in-context setting DiffewS reaches 71.3 one-shot mIoU on COCO-20i and 88.3 on PASCAL-5i, versus 56.1 and 83.2 for SegGPT.
- In the strict few-shot setting DiffewS averages 51.2 one-shot mIoU on COCO-20i, slightly above DCAMA's 50.9; DiffewS-n with training-time N-shot improvements reaches 60.7 5-shot.
- On LVIS-92i out-of-domain, DiffewS scores 31.4 one-shot, outperforming SegGPT (18.6) and other generalists but trailing Matcher (33.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffewS instead of FPTrans | COCO-20^i in-context one-shot segmentation · COCO-20^i | mIoU: 56.5 -> 71.3 | Helped |
| Use DiffewS instead of SegGPT | PASCAL-5^i in-context few-shot segmentation · PASCAL-5^i | mIoU: 89.8 -> 87.8 | Hurt |
| Use DiffewS instead of DCAMA | strict one-shot segmentation · COCO-20^i mean | mIoU: 50.9 -> 51.2 | Helped |
| Use DiffewS instead of DCAMA | strict five-shot segmentation · COCO-20^i mean | mIoU: 58.3 -> 56.0 | Hurt |
| Train with randomly selected 1 to N support samples (ablation) | DiffewS · COCO-20^i strict 5-shot mean | mIoU: 56.0 -> 60.7 | Helped |
| Increase MI2M variance from β¹=(0.00085, 0.012) to β²=(0.0272, 0.384) (ablation) | DiffewS generation process · COCO-20^i fold-0 | mIoU: +23.3% mIoU | Helped |
| Use MI2M instead of MN2M (ablation) | DiffewS generation process · COCO-20^i fold-0 | mIoU: +4.7% mIoU | Helped |
| Use OI2M instead of multi-step generation (ablation) | DiffewS generation process · COCO-20^i fold-0 | mIoU | Helped |
| Use white foreground and black background for query-mask supervision (ablation) | DiffewS · Few-shot segmentation experiments | mIoU | Helped |
| Apply support-mask multiplication in RGB rather than latent space (ablation) | DiffewS multiplication injection · COCO-20^i fold-0 | mIoU: 32.14 -> 33.12 | Helped |
| Use KV fusion rather than QKV fusion (ablation) | DiffewS self-attention · COCO-20^i fold-0 | mIoU: 46.61 -> 46.64 | No clear effect |
| Sample support keys and values during inference (ablation) | DiffewS · COCO in-context 5-shot | mIoU: 72.2 -> 74.7 | Helped |
Surprising findings
- Predicting the mask in one step from the raw query image (OI2M) outperformed multi-step noise-to-mask and image-to-mask generation, suggesting that added noise made the task too easy via information leakage.
- Simple KV fusion self-attention outperformed tokenized cross-attention with CLIP image tokens, and white-on-black mask targets outperformed keeping real-image pixels as targets.
Limitations
- Weaker 5-shot performance than DCAMA; the method was designed mainly for 1-shot and the N-shot extensions give limited gains on PASCAL.
- Fails on large appearance disparity between support and query, look-alike distractors, and heavy occlusion.
- No error bars or variance reporting across seeds.
Experimental setup
Datasets: COCO-20i COCO2014 ADE20K PASCAL VOC PASCAL-5i LVIS-92i
Benchmarks: COCO-20i PASCAL-5i LVIS-92i
Metrics: mIoU
Baselines: HSNet VAT FPTrans Painter SegGPT PerSAM-F Matcher CyCTR BAM DCAMA HDMNet
Builds on: Latent Diffusion Model Marigold DMP SegGPT
Closest papers by meaning
- Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.90)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.89)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- Unlocking the Potential of Pre-trained Vision Transformers for Few-Shot Semantic Segmentation through Relationship Descriptors (CVPR 2024 · similarity 0.89)
- Diffusion-Driven Two-Stage Active Learning for Low-Budget Semantic Segmentation (NeurIPS 2025 · similarity 0.89)
- DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.