Research map / Detection and segmentation / Weakly supervised detection and segmentation
From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S2C, a WSSS framework that transfers SAM's segmentation knowledge into the classifier during training (via prototype contrasting on SAM automatic masks and CAM-prompted SAM self-supervision), improving CAM quality on PASCAL VOC and MS COCO.
Problem
Weakly supervised semantic segmentation from image-level labels suffers from noisy, incomplete CAM seeds; prior SAM-based methods use SAM only at inference, so errors in CAMs propagate.
Approach
Use SAM during classifier training rather than only at inference: SAM-Segment Contrasting pulls pixel features toward prototypes of SAM automatic segments, and the CAM-based Prompting Module turns CAM local peaks into point prompts for SAM, aggregating the resulting class masks with a confidence score (SAM stability times CAM activation) into self-supervision.
Key results
- S2C reaches 76.1% mIoU on PASCAL VOC 2012 train CAMs and 81.7% at the pseudo-label mask level, more than 5% and 6% above prior SoTA.
- S2C achieves 78.2% mIoU on VOC val and 77.5% on VOC test, and 49.8% on MS COCO val, the best among compared WSSS methods.
- Ablation: baseline 50.1% mIoU rises to 57.3% with logit-level SSC, 63.2% with feature-level SSC, and 76.1% with CPM added.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use S2C instead of the strongest listed baseline, CLIP-ES | WSSS CAM generation · PASCAL VOC 2012 train CAMs | mIoU (%): 70.8 -> 76.1 | Helped |
| Use S2C instead of the strongest listed baseline, CLIP-ES | WSSS pseudo-label generation · PASCAL VOC 2012 train pseudo-labels | mIoU (%): 75.0 -> 81.7 | Helped |
| Use S2C instead of MAT | WSSS segmentation model · PASCAL VOC 2012 val | mIoU (%): 73.0 -> 78.2 | Helped |
| Use S2C instead of USAGE | WSSS segmentation model · PASCAL VOC 2012 test | mIoU (%): 72.8 -> 77.5 | Helped |
| Use S2C instead of Xu et al. | WSSS segmentation model · MS-COCO 2014 val | mIoU (%): 45.9 -> 49.8 | Helped |
| Add logit-level SSC (ablation) | Classification-loss-only baseline · PASCAL VOC 2012 train | mIoU (%): 50.1 -> 57.3 | Helped |
| Add feature-level SSC (ablation) | Classification-loss-only baseline · PASCAL VOC 2012 train | mIoU (%): 50.1 -> 63.2 | Helped |
| Add CPM to feature-level SSC (ablation) | S2C classifier · PASCAL VOC 2012 train | mIoU (%): 63.2 -> 76.1 | Helped |
| Use multiple local CAM peaks instead of a single global peak (ablation) | CPM · PASCAL VOC 2012 train | mIoU (%): 73.0 -> 76.1 | Helped |
| Add the SAM stability score to CPM aggregation (ablation) | CPM · PASCAL VOC 2012 train | mIoU (%): 72.7 -> 76.1 | Helped |
| Add the CAM activation score to CPM aggregation (ablation) | CPM · PASCAL VOC 2012 train | mIoU (%): 73.2 -> 76.1 | Helped |
| Add a background score map to CPM (ablation) | CPM · PASCAL VOC 2012 train | mIoU (%): 74.9 -> 76.1 | Helped |
Surprising findings
- Contrasting at the logit level gave a much smaller gain than feature-level contrasting, because SAM segments may span multiple classes or split single objects.
Limitations
- Relies on SAM's automatic segmentation and point prompts, so errors and over-segmentation from SAM propagate into training.
- Comparisons with prior WSSS methods are not fully fair because S2C uses SAM as an additional information source.
Experimental setup
Datasets: PASCAL VOC 2012 MS COCO 2014
Benchmarks: PASCAL VOC 2012 MS COCO 2014
Metrics: mIoU
Baselines: AffinityNet IRNet MCT CLIP-ES MAT USAGE BECO ACR
Builds on: Segment Anything Model ReCo regional prototype contrast SAM-based post-processing InfoNCE
Closest papers by meaning
- SFC: Shared Feature Calibration in Weakly Supervised Semantic Segmentation (AAAI 2024 · similarity 0.90)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.90)
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.90)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.89)
- W2P: Switching from Weak Supervision to Partial Supervision for Semantic Segmentation (AAAI 2024 · similarity 0.89)
- Progressive Feature Self-Reinforcement for Weakly Supervised Semantic Segmentation (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.