Research map / Detection and segmentation / Weakly supervised detection and segmentation
Multi-Label Prototype Visual Spatial Search for Weakly Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MuP-VSS, a ViT-based weakly supervised semantic segmentation method that learns multi-label prototypes via global embedding and a Prototype Embedding Module, optimized by cross-class, cross-image, and patch-to-prototype losses to reduce classification bias in image-level WSSS.
Problem
Image-level weakly supervised semantic segmentation produces incomplete pseudo masks from CAM and transformer self-attention maps because the multi-label classification loss focuses on discriminative regions (classification bias).
Approach
Replaces classification-driven localization with query-based visual spatial search: learned multi-label prototypes (from a convolutional global embedding) are matched against patch tokens, and they are optimized with contrastive losses exploiting exclusivity (cross-class) and consistency (cross-image) plus a patch-to-prototype consistency loss, instead of the multi-label classification loss on pro
Key results
- Seed mIoU on VOC train of 71.7% vs. 69.5% for the second-best CTI with DeiT-S; pseudo-mask mIoU 74.1%.
- Segmentation on VOC val 73.6% and test 74.7% mIoU with ResNet38 DeepLab, versus 72.2% for DuPL on val.
- On MS COCO val, 46.6% mIoU, outperforming DiG (45.5%) with ResNet38.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MuP-VSS instead of CTI | DeiT-S WSSS seed generation · PASCAL VOC 2012 train (Seed) | mIoU (%): 69.5 -> 71.7 | Helped |
| use MuP-VSS instead of CTI | DeiT-S WSSS pseudo-mask generation · PASCAL VOC 2012 train (Mask) | mIoU (%): 73.7 -> 74.1 | Helped |
| use MuP-VSS instead of DuPL | semantic segmentation · PASCAL VOC 2012 val | mIoU (%): 72.2 -> 73.6 | Helped |
| use MuP-VSS instead of AEFT | semantic segmentation · PASCAL VOC 2012 test | mIoU (%): 71.7 -> 74.7 | Helped |
| use MuP-VSS instead of DiG | semantic segmentation · MS COCO val | mIoU (%): 45.5% vs 46.6% | Helped |
| remove multi-label prototype optimization (MPO) (ablation) | MuP-VSS · PASCAL VOC 2012 train (Seed) | mIoU (%): 71.7 -> 64.2 | Hurt |
| remove multi-label prototype representation (MPR) (ablation) | MuP-VSS · PASCAL VOC 2012 train (Seed) | mIoU (%): 71.7 -> 63.1 | Hurt |
| add Global Embedding (ablation) | Base-R prototype representation · PASCAL VOC 2012 train (Seed) | mIoU (%): 63.1 -> 67.1 | Helped |
| add PEM (ablation) | Base-R prototype representation · PASCAL VOC 2012 train (Seed) | mIoU (%): 63.1 -> 66.6 | Helped |
| add CCP loss (ablation) | Base-O prototype optimization · PASCAL VOC 2012 train (Seed) | mIoU (%): 64.2 -> 67.5 | Helped |
| add CIP loss (ablation) | Base-O prototype optimization · PASCAL VOC 2012 train (Seed) | mIoU (%): 64.2 -> 66.7 | Helped |
| add P2P loss (ablation) | Base-O prototype optimization · PASCAL VOC 2012 train (Seed) | mIoU (%): 64.2 -> 68.4 | Helped |
Limitations
- Relies on a pre-set empirical threshold (tau_PEM, tau_p2p) for filtering affinity and response maps.
- Pseudo-mask results depend on PSA post-processing and the DeepLab-ResNet38 segmentation network for final numbers.
Experimental setup
Datasets: PASCAL VOC 2012 MS COCO 2014
Benchmarks: PASCAL VOC 2012 MS COCO 2014
Metrics: mIoU
Baselines: CDA ReCAM CLIMS ACR D2CAM SFC KTSE MCTformer CTI DiG DuPL DIAL
Builds on: Class Activation Map MCTformer ViT PSA DeepLab
Closest papers by meaning
- PSDPM: Prototype-based Secondary Discriminative Pixels Mining for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.88)
- Class Tokens Infusion for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
- Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP (AAAI 2025 · similarity 0.87)
- WISH: Weakly Supervised Instance Segmentation using Heterogeneous Labels (CVPR 2025 · similarity 0.87)
- Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
- W2P: Switching from Weak Supervision to Partial Supervision for Semantic Segmentation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.