Research map / Detection and segmentation / Weakly supervised detection and segmentation
Extreme Point Supervised Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EXITS, a weakly supervised instance segmentation method that uses free extreme-point annotations (topmost, leftmost, bottommost, rightmost) propagated through a ViT-based random-walk label propagation to train a pseudo-label generator, whose masks then supervise any instance segmentation model.
Problem
Box-supervised instance segmentation relies on the bounding box tightness prior, which fails under occlusion and multi-part objects; extreme points come free with box annotation but are underused for mask learning.
Approach
Treat extreme points as true foreground, propagate them over a fully connected graph of points using ViT self-attention-derived transition probabilities (Sinkhorn-normalized random walk) to retrieve pseudo foreground/background points, and train a per-object pseudo-label generator with point and CRF losses before training the final instance segmenter on its masks.
Key results
- On COCO test-dev with ResNet-101-DCN and SOLOv2, EXITS reaches 40.2 Mask AP vs. 38.7 for MAL (+1.5).
- On PASCAL VOC val2012 with ResNet-50, EXITS achieves 40.4 AP vs. 38.6 for BoxTeacher and 37.6 for MAL.
- On Separated COCO, pseudo-label mIoU improves from 59.3 (MAL) to 66.6 (EXITS), a 7.3 point gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EXITS extreme-point supervision instead of box supervision | ResNet-50 + SOLOv2 · PASCAL VOC val2012 | AP: 38.6 -> 40.4 | Helped |
| use EXITS extreme-point supervision instead of box supervision | ResNet-101 + SOLOv2 · PASCAL VOC val2012 | AP: 40.3 -> 41.4 | Helped |
| use EXITS instead of MAL | ResNet-101-DCN + SOLOv2 · COCO val2017 | Mask AP: 38.2 -> 39.8 | Helped |
| use EXITS instead of MAL | Swin-Small + Mask2Former · COCO test-dev | Mask AP: 44.1 -> 45.0 | Helped |
| use EXITS instead of MAL | ResNeXt-101-32 + Mask R-CNN · LVIS v1.0 val | Mask AP: 23.7 -> 24.1 | Helped |
| use EXITS pseudo-label generation instead of MAL | pseudo label generator · Separated COCO | mIoU: 59.3 -> 66.6 | Helped |
| add initial extreme-point and background point supervision (ablation) | EXITS point loss · PASCAL VOC val2012 | AP: 37.6 -> 38.0 | Helped |
| add retrieved pseudo-point supervision (ablation) | EXITS point loss with initial point supervision · PASCAL VOC val2012 | AP: 38.0 -> 40.4 | Helped |
| use point dropout (ablation) | EXITS pseudo-point supervision · PASCAL VOC val2012 | AP: 39.8 -> 40.4 | Helped |
Surprising findings
- Naively using extreme points as point labels gives only marginal gain (37.6 to 38.0 AP); the gain comes mainly from the propagated pseudo points.
Limitations
- Differentiating between occluded objects of the same class remains challenging.
- Pipeline is multi-stage: the pseudo-label generator must be trained before the final model.
- Extreme point annotation is assumed to be available; results for box-only datasets are not applicable.
Experimental setup
Datasets: COCO PASCAL VOC LVIS v1.0 Separated COCO
Benchmarks: COCO val2017 COCO test-dev PASCAL VOC val2012 LVIS v1.0 Separated COCO
Metrics: Mask AP retention rate mIoU
Baselines: MAL BoxTeacher SIM BoxLevelSet DiscoBox BoxInst SOLOv2 Mask2Former Mask R-CNN
Builds on: MAL BoxInst DEXTR Mask Auto Labeler SOLOv2
Closest papers by meaning
- Pointly-Supervised Instance Segmentation (CVPR 2022 · similarity 0.91)
- Sketchy Bounding-box Supervision for 3D Instance Segmentation (CVPR 2025 · similarity 0.89)
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.88)
- WISH: Weakly Supervised Instance Segmentation using Heterogeneous Labels (CVPR 2025 · similarity 0.88)
- MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving (AAAI 2024 · similarity 0.87)
- P2Seg: Pointly-supervised Segmentation via Mutual Distillation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.