Research map / Detection and segmentation / Open-set and few-shot detection
MaskDiff: Modeling Mask Distribution with Diffusion Probabilistic Model for Few-Shot Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaskDiff, which models instance masks as a conditional diffusion distribution (conditioned on object region and K-shot information) instead of a point-estimate prototype, for few-shot instance segmentation on COCO.
Problem
Few-shot instance segmentation must segment novel-category instances from a query image given only K annotated examples; prototype-based point estimates (e.g., mean of K-shot features) limit mask prediction quality.
Approach
Model the binary mask distribution with a conditional denoising diffusion probabilistic model conditioned on the object region and K-shot information, trained with a two-stage schedule (detector first, then mask diffusion head) and sampled with classifier-free guidance using a one-hot class vector.
Key results
- On COCO-Novel 1-shot, MaskDiff reaches 6.26 segmentation AP versus 5.56 for iFS-RCNN's 5.56 (detection AP 7.13 vs 6.37).
- On COCO-All 10-shot, MaskDiff achieves 31.89 segmentation AP versus 30.41 for iFS-RCNN.
- MaskDiff runs at 8.27 FPS versus 10.28 FPS for iFS-RCNN, trading speed for accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add classifier-free guided mask sampling (ablation) | MaskDiff · COCO-Novel, 1-shot | segmentation AP: 6.07 -> 6.26 | Helped |
| add classifier-free guided mask sampling (ablation) | MaskDiff · COCO-Novel, 5-shot | segmentation AP: 9.74 -> 10.09 | Helped |
| add classifier-free guided mask sampling (ablation) | MaskDiff · COCO-Novel, 10-shot | segmentation AP: 10.76 -> 11.01 | Helped |
| use two-stage training (ablation) | MaskDiff · COCO-Novel, 1-shot | segmentation AP: 5.13 -> 6.26 | Helped |
| use two-stage training (ablation) | MaskDiff · COCO-Novel, 5-shot | segmentation AP: 8.91 -> 10.09 | Helped |
| use two-stage training (ablation) | MaskDiff · COCO-Novel, 10-shot | segmentation AP: 10.02 -> 11.01 | Helped |
| increase diffusion steps from 250 to 1000 (ablation) | MaskDiff · COCO-All, 1-shot | AP, all classes: 29.26 -> 29.42 | Helped |
Surprising findings
- Removing the two-stage training drops novel-class segmentation AP drastically (1-shot 6.26 to 4.92 without guidance and two-stage; 5.13 without guidance-only variants), showing detector pretraining matters a lot.
Limitations
- Slower inference than conventional CNN-based FSIS methods due to iterative diffusion sampling (1000 steps)
- Performance drops with fewer diffusion steps
- Failures from misclassification and imprecise masks on novel classes
Experimental setup
Datasets: COCO
Benchmarks: COCO-All COCO-Novel COCO-20i
Metrics: AP AP50 AP75 AP95 FPS
Baselines: ONCE TFA FSDetView LEAST MRCN+ft-full MTFA iMTFA iFS-RCNN Meta-RCNN FAPIS DTN
Builds on: TFA Mask R-CNN classifier-free guidance DDPM
Closest papers by meaning
- Mask Encoding for Single Shot Instance Segmentation (CVPR 2020 · similarity 0.90)
- Mask Matching Transformer for Few-Shot Segmentation (NeurIPS 2022 · similarity 0.90)
- ContrastMask: Contrastive Learning To Segment Every Thing (CVPR 2022 · similarity 0.89)
- Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (CVPR 2020 · similarity 0.88)
- SOLOv2: Dynamic and Fast Instance Segmentation (NeurIPS 2020 · similarity 0.88)
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.