Research map / Detection and segmentation / Weakly supervised detection and segmentation
POT: Prototypical Optimal Transport for Weakly Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prototypical Optimal Transport (POT), a plug-and-play framework that splits confident features into multiple clusters, allocates features with similarity-aware optimal transport, and adds an OT-based consistency loss to make weakly supervised segmentation CAMs more complete.
Problem
Classifier CAMs in weakly supervised semantic segmentation highlight only discriminative regions, and existing prototype-based methods use a single prototype per class, which cannot capture intra-class variation and yields incomplete CAMs.
Approach
Use multiple cluster prototypes per class, assign features to clusters via optimal transport whose target marginal is set by similarity between cluster prototypes and the classifier prototype (instead of uniform), reweight cluster-activated CAMs by the transport plan, and supervise with a pixel-adaptive OT-based consistency loss.
Key results
- On PASCAL VOC 2012, CLIP-ES + Ours reaches 76.1% mIoU on val and 76.7% on test, state-of-the-art among multistage image-level WSSS methods.
- On MS COCO 2014 val, CLIP-ES + Ours achieves 47.9% mIoU, up from 45.4% for CLIP-ES.
- Ablation on VOC train: multi-cluster activation raises CLIP-ES from 70.8% to 72.9% CAM mIoU, and full POT (with SOT and OT consistency loss) reaches 75.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add POT to CLIP-ES | CLIP-ES · PASCAL VOC 2012 val and test; MS COCO 2014 val | mIoU (%): 71.1 -> 76.1; 71.4 -> 76.7; 45.4 -> 47.9 | Helped |
| Add POT to SFC | SFC · PASCAL VOC 2012 val and test; MS COCO 2014 val | mIoU (%): 71.2 -> 74.5; 72.5 -> 75.2; 46.8 -> 47.6 | Helped |
| Add multi-cluster activation instead of single-cluster activation (ablation) | CLIP-ES distillation baseline · PASCAL VOC 2012 train | mIoU (%): 71.5 -> 72.9 (+1.4%) | Helped |
| Add vanilla OT feature allocation (ablation) | Multi-cluster activation baseline · PASCAL VOC 2012 train | mIoU (%): 72.9 -> 74.1 | Helped |
| Use similarity-aware OT instead of vanilla OT (ablation) | Multi-cluster activation · PASCAL VOC 2012 train | mIoU (%): 74.1 -> 74.6 (+0.5%) | Helped |
| Add OT-based consistency loss (ablation) | Similarity-aware OT · PASCAL VOC 2012 train | mIoU (%): 74.6 -> 75.0 | Helped |
| Use adaptive pixel weights in the consistency loss (ablation) | Similarity-aware OT with consistency loss · PASCAL VOC 2012 train | mIoU (%): 74.6 -> 75.0 | Helped |
Limitations
- Relies on a fixed number of clusters K (set to 3) and a confidence threshold tau (0.7) chosen as hyperparameters.
- Improvements are reported mainly on CAM and pseudo-label quality, with ablations run on a distilled ResNet-50 rather than the full pipeline.
Experimental setup
Datasets: PASCAL VOC 2012 SBD MS COCO 2014
Benchmarks: PASCAL VOC 2012 MS COCO 2014
Metrics: mIoU
Baselines: CLIP-ES SFC SIPE CLIP-ES+PSDPM CLIP-ES+CPAL MCTformer AMN LPCAM AdvCAM CLIMS
Builds on: CLIP-ES SFC Sinkhorn optimal transport LPCAM
Closest papers by meaning
- [Re] Background-Aware Pooling and Noise-Aware Loss for Weakly-Supervised Semantic Segmentation (NeurIPS 2022 · similarity 0.90)
- Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast (CVPR 2022 · similarity 0.88)
- Simple Does It: Weakly Supervised Instance and Semantic Segmentation (CVPR 2017 · similarity 0.88)
- Combining Bottom-Up, Top-Down, and Smoothness Cues for Weakly Supervised Image Segmentation (CVPR 2017 · similarity 0.88)
- Progressive Feature Self-Reinforcement for Weakly Supervised Semantic Segmentation (AAAI 2024 · similarity 0.86)
- WISH: Weakly Supervised Instance Segmentation using Heterogeneous Labels (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.