Research map / Label-efficient and robust learning / Recognition and long-tail
SFC: Shared Feature Calibration in Weakly Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Shared Feature Calibration (SFC), which uses a multi-scaled distribution-weighted consistency loss between classifier-weight CAMs and prototype CAMs plus image bank re-sampling to fix long-tail-induced over/under-activation in CAMs for image-level WSSS.
Problem
Image-level weakly supervised semantic segmentation relies on CAMs, which become over-activated for head classes and under-activated for tail classes under the long-tailed distribution of training data due to shared features, degrading pseudo-label quality.
Approach
Identify that shared features in classifier weights bias CAMs under long-tailed data, and calibrate them by enforcing consistency between classifier-weight CAMs and prototype CAMs, re-weighted per class by sample-count gaps (DC coefficient), together with image bank re-sampling to boost tail-class frequency.
Key results
- Pseudo-label (final mask) mIoU of 73.7% on PASCAL VOC 2012 training set, 1.5% above AMN (72.2%).
- Segmentation mIoU of 71.2% (val) and 72.5% (test) on PASCAL VOC 2012 with ResNet101 backbone.
- 46.8% mIoU on MS COCO 2014 validation, outperforming AMN by 2.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SFC CAM calibration | PASCAL VOC 2012 training-set CAM generation · PASCAL VOC 2012 training set CAM | mIoU (%): 62.1 -> 64.7 | Helped |
| add SFC CAM calibration | PASCAL VOC 2012 training-set CRF refinement · PASCAL VOC 2012 training set CRF | mIoU (%): 65.3 -> 69.4 | Helped |
| add SFC pseudo-label generation | PASCAL VOC 2012 training-set mask generation · PASCAL VOC 2012 training set Mask | mIoU (%): 72.2 -> 73.7 | Helped |
| use SFC instead of AMN | ResNet101 WSSS · PASCAL VOC 2012 val | mIoU: 70.7 -> 71.2 | Helped |
| use SFC instead of EPS | ResNet101 WSSS · PASCAL VOC 2012 test | mIoU: 71.8 -> 72.5 | Helped |
| use SFC instead of AMN | ResNet101 WSSS · MS COCO 2014 val | mIoU: 44.7 -> 46.8 | Helped |
| add IBR to Base (ablation) | Base classifier-weight CAM · PASCAL VOC 2012 training set CAM | mIoU (%): 55.1 -> 55.9 | Helped |
| remove IBR from SFC (ablation) | SFC CAM generation · PASCAL VOC 2012 training set | mIoU (%): 62.4 -> 64.7 | Hurt |
| add classifier-weight multi-scale consistency loss (ablation) | IBR + prototype consistency · PASCAL VOC 2012 training set | mIoU (%): 62.4 -> 64.7 | Helped |
| use classifier-weight consistency loss alone, without prototype consistency (ablation) | IBR-based SFC · PASCAL VOC 2012 training set | mIoU (%): 58.1 -> 64.7 | Hurt |
| remove DC from both consistency losses (ablation) | SFC consistency losses · PASCAL VOC 2012 training set | mIoU (%): 62.0 -> 64.7 | Hurt |
| remove DC from classifier-weight consistency loss (ablation) | SFC consistency losses · PASCAL VOC 2012 training set | mIoU (%): 63.7 -> 64.7 | Hurt |
Surprising findings
- Using the consistency loss alone without image bank re-sampling yields lower gains than the full combination, and using the down-scaled weight-CAM loss alone drops performance significantly (58.1% vs 55.1% base mIoU in setting IV is an improvement, but far below full model).
Limitations
- Consistency loss can hurt when head and tail class sample counts are approximately balanced, so the DC re-weighting is needed to avoid negative effects.
- Relies on a long-tailed-distribution assumption; the analysis is for simplified one-head/one-tail case.
Experimental setup
Datasets: PASCAL VOC 2012 SBD (augmented) MS COCO 2014 ImageNet
Benchmarks: PASCAL VOC 2012 MS COCO 2014
Metrics: mIoU
Baselines: IRN SEAM CONTA AdvCAM RIB CLIMS ESOL SIPE AMN
Builds on: IRN AMN CRF
Closest papers by meaning
- Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach (AAAI 2020 · similarity 0.90)
- From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Single-Stage Semantic Segmentation From Image Labels (CVPR 2020 · similarity 0.89)
- DuPL: Dual Student with Trustworthy Progressive Learning for Robust Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
- W2P: Switching from Weak Supervision to Partial Supervision for Semantic Segmentation (AAAI 2024 · similarity 0.87)
- PSDPM: Prototype-based Secondary Discriminative Pixels Mining for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.