Research map / Detection and segmentation / Weakly supervised detection and segmentation
Scribble Hides Class: Promoting Scribble-Based Weakly-Supervised Semantic Segmentation with Its Class Label
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a class-driven scribble promotion network that adds image-level class-based CAM pseudo-labels to scribble supervision, with a localization rectification module and distance entropy loss, reaching SOTA on ScribbleSup.
Problem
Scribble-based weakly-supervised semantic segmentation relies on sparse scribbles; label-diffusion pseudo-labels use only local cues and degrade when scribbles are shrunk or dropped.
Approach
Extract image-level class labels from scribbles to train a classifier that yields globally considered CAM pseudo-labels, combine them with scribble supervision, rectify foreground features via a localization rectification module, and weight predictions with a distance entropy loss based on distances to scribbles and pseudo-label boundaries.
Key results
- Best configuration (ResNet-101 + DeepLabV3+) achieves 75.9% mIoU on ScribbleSup val and 76.0% on test, outperforming TEL (75.2 val / 75.6 test) and AGMM (74.2 / 75.7).
- Ablation with DeepLabV2 + ResNet-50: combining scribble and pseudo-label supervision gives 72.13% mIoU versus about 66-67% for either alone; full model reaches 73.91%.
- Under scribble shrink to points, AGMM and TEL degrade by about 10%, while the proposed method drops by less than 1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CDSP instead of TEL | ResNet101 + DeepLabV3+ on ScribbleSup · ScribbleSup val | mIoU: 75.2 -> 75.9 | Helped |
| Use CDSP instead of AGMM | ResNet101 + DeepLabV3+ on ScribbleSup · ScribbleSup test | mIoU: 75.7 -> 76.0 | Helped |
| Add pseudo-label supervision to scribble supervision (ablation) | CDSP on ScribbleSup · ScribbleSup val | mIoU: 66.17 -> 72.13 | Helped |
| Add pseudo-label-boundary distance entropy loss to basic supervision (ablation) | CDSP on ScribbleSup · ScribbleSup val | mIoU: 72.13 -> 67.33 | Hurt |
| Add scribble distance entropy loss to basic supervision (ablation) | CDSP on ScribbleSup · ScribbleSup val | mIoU: 72.13 -> 73.38 | Helped |
| Add both distance entropy terms to basic supervision (ablation) | CDSP on ScribbleSup · ScribbleSup val | mIoU: 72.13 -> 73.58 | Helped |
| Add LoRM to basic supervision (ablation) | CDSP on ScribbleSup · ScribbleSup val | mIoU: 72.13 -> 73.26 | Helped |
| Add LoRM to basic supervision with pseudo-label-boundary distance entropy (ablation) | CDSP on ScribbleSup · ScribbleSup val | mIoU: 67.33 -> 73.64 | Helped |
| Add LoRM and both distance entropy terms (ablation) | CDSP on ScribbleSup · ScribbleSup val | mIoU: 72.13 -> 73.91 | Helped |
| Replace SEAM pseudo-labels with AFA pseudo-labels (ablation) | CDSP on ScribbleSup · ScribbleSup with ResNet50 | mIoU: 69.8 -> 71.5 | Helped |
| Replace AFA pseudo-labels with BMP pseudo-labels (ablation) | CDSP on ScribbleSup · ScribbleSup with ResNet50 | mIoU: 71.5 -> 73.9 | Helped |
| Replace SEAM pseudo-labels with AFA pseudo-labels (ablation) | CDSP on ScribbleSup · ScribbleSup with ResNet101 | mIoU: 71.8 -> 73.3 | Helped |
Surprising findings
- Adding the distance entropy loss on the pseudo-label boundary alone (L_dc) does not improve over basic supervision, attributed to overfitting noisy pseudo-labels; LoRM resolves this.
Limitations
- Relies on pseudo-labels from an image-level WSSS method (BMP), so segmentation quality depends on pseudo-label accuracy.
- Noisy pseudo-labels near object boundaries and nearby objects can still mislead the model, requiring LoRM and DEL.
Experimental setup
Datasets: ScribbleSup (PASCAL VOC2012 + SBD with scribbles) scribble-shrink scribble-drop
Benchmarks: PASCAL VOC2012 ScribbleSup val PASCAL VOC2012 test
Metrics: mIoU
Baselines: ScribbleSup RAWKS NCL KCL BPG PSI URSS CCL TEL AGMM
Builds on: class activation map BMP URSS DeepLabV3+ label smoothing
Closest papers by meaning
- Scribbles for All: Benchmarking Scribble Supervised Segmentation Across Datasets (NeurIPS 2024 · similarity 0.93)
- ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation (CVPR 2016 · similarity 0.92)
- Scribble-Supervised Semantic Segmentation with Prototype-based Feature Augmentation (ICML 2024 · similarity 0.91)
- Exploratory Inference Learning for Scribble Supervised Semantic Segmentation (AAAI 2023 · similarity 0.90)
- Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
- W2P: Switching from Weak Supervision to Partial Supervision for Semantic Segmentation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.