Research map / Detection and segmentation / Weakly supervised detection and segmentation
Scribble-Supervised Semantic Segmentation with Prototype-based Feature Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes prototype-based feature augmentation (local and global feature prototypes from high-confidence pixels) for scribble-supervised semantic segmentation, reaching 77.9% mIoU on PASCAL VOC 2012 with MiT-B1.
Problem
Scribble-supervised semantic segmentation propagates sparse scribble labels to unlabeled pixels, but existing methods ignore features of already correctly classified pixels during propagation.
Approach
Extract class prototypes from high-confidence pixel features of the current image (local) and maintain a memory bank of multiple prototypes per class (global), then use attention-based prototype feature augmenters with a consistency loss between original and augmented predictions, adapting FeatMatch-style augmentation from image-level/semi-supervised settings to pixel-level scribble supervision.
Key results
- Ours-MiT-B1 achieves 77.9% mIoU on PASCAL VOC 2012 val with scribble supervision, exceeding TEL (77.3%) by 0.6 points.
- Ablation: partial cross-entropy baseline 67.5% mIoU rises to 77.9% with both local and global prototype augmentation (+10.4).
- With MiT-B5 backbone, the method reaches 81.5% mIoU (excluded from main comparison for fairness).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add local prototype augmentation (consistency loss) to partial CE baseline (ablation) | MiT-B1 scribble segmentation · PASCAL VOC 2012 val | mIoU: +9.4 mIoU (67.5 to 76.9) | Helped |
| add global prototype augmentation (consistency loss) to partial CE baseline (ablation) | MiT-B1 scribble segmentation · PASCAL VOC 2012 val | mIoU: +9.9 mIoU (67.5 to 77.4) | Helped |
| combine local and global prototype augmentation (ablation) | MiT-B1 scribble segmentation · PASCAL VOC 2012 val | mIoU: +10.4 mIoU over baseline; +0.5 over global only | Helped |
| use global-prototype-augmented prediction under partial CE vs other loss choice (ablation) | two-prototype variant · PASCAL VOC 2012 val | mIoU | No clear effect |
| proposed method vs TEL state of the art on scribble supervision with comparable backbone | MiT-B1 · PASCAL VOC 2012 val | mIoU: +0.6 (77.3 to 77.9) | Helped |
| increase number of global prototypes per class (ablation) | global prototype augmentation only · PASCAL VOC 2012 val | mIoU: gains saturate around 5 prototypes per class | Helped |
| top-k percentage for prototype extraction set to 8% vs other values (ablation) | local prototype augmentation · PASCAL VOC 2012 val | mIoU | Mixed |
| scale backbone from MiT-B1 to MiT-B5 or MiT-B3 under scribble supervision (ablation) | Segformer with proposed method · PASCAL VOC 2012 val | mIoU: MiT-B3 79.8, MiT-B5 81.5 vs MiT-B1 77.9 | Helped |
Surprising findings
- Using only global prototypes with partial cross-entropy guidance gives the best single-branch result, and combining both prototype types gives the top overall result.
Limitations
- Comparison fairness restricted to MiT-B1 backbone; gains on larger backbones not included in main table.
- Consistency loss is described as MSE in text but formula is cross-entropy-like, and the method relies on heuristic hyperparameters (top-k 8%, 5 global prototypes, alpha 0.99).
Experimental setup
Datasets: PASCAL-Scribble PASCAL VOC 2012 ImageNet
Benchmarks: PASCAL VOC 2012
Metrics: mIoU
Baselines: TEL AGMM CDSP URSS A2GNN SPML BPG GatedCRF
Builds on: FeatMatch SegFormer ScribbleSup
Closest papers by meaning
- Scribbles for All: Benchmarking Scribble Supervised Segmentation Across Datasets (NeurIPS 2024 · similarity 0.92)
- Scribble Hides Class: Promoting Scribble-Based Weakly-Supervised Semantic Segmentation with Its Class Label (AAAI 2024 · similarity 0.91)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.85)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.83)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.83)
- Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.