Research map / Detection and segmentation / Weakly supervised detection and segmentation
PSDPM: Prototype-based Secondary Discriminative Pixels Mining for Weakly Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PSDPM, a plug-and-play framework that uses a self-attention-based Foreground Pixel Estimation Module and a consistency loss to pull prototype CAMs toward estimated secondary discriminative foreground pixels, improving WSSS pseudo labels and segmentation on PASCAL VOC 2012 and MS COCO 2014.
Problem
Image-level weakly supervised semantic segmentation produces incomplete CAMs because class prototypes, learned under classification loss, only activate primary discriminative pixels and miss secondary discriminative pixels.
Approach
Estimate potential foreground pixels with a cascaded self-attention module (FPEM) over multi-level features, RGB and position, fused with the DeepLab output and prototype CAM, and enforce a consistency loss that aligns the prototype CAM with this estimate so the classifier learns secondary discriminative features.
Key results
- CLIP-ES + PSDPM reaches 74.1% mIoU on PASCAL VOC 2012 val and 74.9% on test with DeepLab V2, state of the art among image-level WSSS methods.
- PSDPM on CLIP-ES reaches 47.2% mIoU on MS COCO 2014 val (+1.8 over CLIP-ES).
- PSDPM improves CLIMS pseudo-label quality from 70.5% to 75.5% mIoU on PASCAL VOC 2012 training set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PSDPM to CLIP-ES | CLIP-ES · PASCAL VOC 2012 val | mIoU: 71.1 -> 74.1 | Helped |
| Add PSDPM to CLIP-ES | CLIP-ES · PASCAL VOC 2012 test | mIoU: 71.4 -> 74.9 | Helped |
| Add PSDPM to CLIP-ES | CLIP-ES · MS COCO 2014 val | mIoU: 45.4 -> 47.2 | Helped |
| Add prototype CAM to DeepLab V2 output (ablation) | PSDPM with CLIMS baseline · PASCAL VOC 2012 train | mIoU: 59.7 -> 61.3 | Helped |
| Add FPEM to the DeepLab V2 output and prototype CAM combination (ablation) | PSDPM with CLIMS baseline · PASCAL VOC 2012 train | mIoU: 61.3 -> 68.5 | Helped |
| Add prototype CAM to the DeepLab V2 output used with FPEM (ablation) | PSDPM with CLIMS baseline · PASCAL VOC 2012 train | mIoU: 67.9 -> 68.5 | Helped |
| Use Mr3 instead of no FPEM for foreground pixel estimation (ablation) | PSDPM with CLIMS baseline · PASCAL VOC 2012 train | mIoU: 61.3 -> 64.8 | Helped |
| Continue FPEM refinement from Mr3 to Mr2 (ablation) | PSDPM with CLIMS baseline · PASCAL VOC 2012 train | mIoU: 64.8 -> 68.0 | Helped |
| Continue FPEM refinement from Mr2 to Mr1 (ablation) | PSDPM with CLIMS baseline · PASCAL VOC 2012 train | mIoU: 68.0 -> 68.5 | Helped |
| Remove the position matrix from FPEM input (ablation) | PSDPM with CLIMS baseline · PASCAL VOC 2012 train | mIoU: 68.5 -> 68.1 | Hurt |
| Remove extracted feature maps from FPEM input (ablation) | PSDPM with CLIMS baseline · PASCAL VOC 2012 train | mIoU: 68.5 -> 64.2 | Hurt |
| Remove RGB image from FPEM input (ablation) | PSDPM with CLIMS baseline · PASCAL VOC 2012 train | mIoU: 68.5 -> 68.2 | Hurt |
Surprising findings
- Simply adding post-processing (IRN) to CLIP-ES barely changes pseudo-label mIoU (75.0 to 75.1), suggesting gains are not from post-processing.
Limitations
- Prototype CAM resolution is low, requiring post-processing such as IRN or DenseCRF to refine boundaries.
- Gains depend on the quality of the baseline method's seed pixels from the DeepLab output.
Experimental setup
Datasets: PASCAL VOC 2012 SBD MS COCO 2014
Benchmarks: PASCAL VOC 2012 MS COCO 2014
Metrics: mIoU
Baselines: IRN SEAM AdvCAM RIB AMN SIPE ESOL CLIMS LPCAM CLIP-ES MCTformer ToCo
Builds on: CLIMS SIPE IRN CLIP-ES LPCAM
Closest papers by meaning
- From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.88)
- Multi-Label Prototype Visual Spatial Search for Weakly Supervised Semantic Segmentation (CVPR 2025 · similarity 0.88)
- SFC: Shared Feature Calibration in Weakly Supervised Semantic Segmentation (AAAI 2024 · similarity 0.87)
- DuPL: Dual Student with Trustworthy Progressive Learning for Robust Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
- Progressive Feature Self-Reinforcement for Weakly Supervised Semantic Segmentation (AAAI 2024 · similarity 0.87)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.