Research map / Detection and segmentation / Weakly supervised detection and segmentation
WISH: Weakly Supervised Instance Segmentation using Heterogeneous Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WISH, a weakly supervised instance segmentation framework that unifies heterogeneous weak labels (image tags, points, boxes) via SAM's prompt latent space and multi-stage matching, and studies cost-constrained label mixes on Pascal VOC and COCO.
Problem
Weakly supervised instance segmentation typically uses one weak label type per dataset, ignoring the cost-performance benefits of mixing heterogeneous weak labels (class tags, points, boxes) under a fixed annotation budget.
Approach
Treat weak labels as SAM prompts mapped into SAM's prompt latent space; a prompt prediction head is matched to GT prompt latents (KL divergence) alongside class and SAM-mask costs in a Mask2Former-based Hungarian matching, with class tags converted to point prompts from CAM local peaks.
Key results
- Under homogeneous settings on VOC, WISH reaches 46.0 AP with class tags, 52.4 with points, and 54.6 with boxes, exceeding prior SoTA in each setting.
- On COCO val, WISH achieves 20.7 AP (tags), 31.9 AP (points), and 42.3 AP (boxes).
- On VOC with a fixed tag-budget, a tag-box mix (5,290 tags + 441 boxes... specifically 5,290 tags with 441 boxes) reaches 48.3 AP versus 46.0 for tags only and 45.1 for boxes only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a heterogeneous mix of tags and boxes instead of tags only under the fixed annotation budget | WISH · PASCAL VOC 2012 val | AP: 46.0 -> 48.3 | Helped |
| Use a heterogeneous mix of tags and boxes instead of boxes only under the fixed annotation budget | WISH · PASCAL VOC 2012 val | AP: 45.1 -> 48.3 | Helped |
| Add prompt matching loss Lprompt (ablation) | WISH · PASCAL VOC, class tags | mAP: 43.6 -> 46.0 | Helped |
| Use adaptive selection among SAM mask levels rather than only level 1 (ablation) | WISH · PASCAL VOC, class tags | mAP: 36.2 -> 46.0 | Helped |
| Use adaptive selection among SAM mask levels rather than only level 2 (ablation) | WISH · PASCAL VOC, class tags | mAP: 40.7 -> 46.0 | Helped |
| Use adaptive selection among SAM mask levels rather than only level 3 (ablation) | WISH · PASCAL VOC, class tags | mAP: 42.8 -> 46.0 | Helped |
| Add self-enhancement loss Lself (ablation) | WISH · PASCAL VOC, class tags | mAP: 41.5 -> 46.0 | Helped |
| Predict masks with the prompt head followed by the SAM decoder rather than the WISH mask head (ablation) | WISH · PASCAL VOC | mAP: over a 3% decrease in mAP | Hurt |
Surprising findings
- Using the SAM mask decoder directly with the prompt head drops mAP by over 3% compared with the dedicated mask head.
- Heterogeneous tag-box mixing at equal cost outperforms both homogeneous extremes by about 2 AP on VOC.
Limitations
- Does not account for image acquisition cost or use unlabeled images in a semi-supervised manner.
- Cost ratios (1:6:12) are estimated from AMT pricing and may vary by dataset and application.
- Only tag-point and tag-box combinations were explored due to computational cost; the full combination space was not tested.
Experimental setup
Datasets: PASCAL VOC 2012 augmented MS-COCO 2017
Benchmarks: PASCAL VOC 2012 val COCO 2017 val COCO 2017 test-dev
Metrics: mask AP AP50 AP75
Baselines: Mask2Former IRN BESTIE Attnshift SAPNet BoxInst DiscoBox BoxLevelSet SIM BoxTeacher MAL SOLOv2
Builds on: Mask2Former SAM BESTIE BoxTeacher
Closest papers by meaning
- Simple Does It: Weakly Supervised Instance and Semantic Segmentation (CVPR 2017 · similarity 0.91)
- Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement (CVPR 2022 · similarity 0.91)
- P2Seg: Pointly-supervised Segmentation via Mutual Distillation (ICLR 2024 · similarity 0.88)
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.88)
- Extreme Point Supervised Instance Segmentation (CVPR 2024 · similarity 0.88)
- MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.