Research map / Detection and segmentation / Weakly supervised detection and segmentation
Sketchy Bounding-box Supervision for 3D Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sketchy-3DIS, a weakly supervised 3D instance segmentation framework that learns from perturbed (sketchy) bounding boxes via an adaptive box-to-point pseudo labeler jointly trained with a coarse-to-fine query-based instance segmentator.
Problem
Bounding-box supervised 3D instance segmentation assumes accurate, compact boxes, which are hard to obtain in practice; existing box-supervised methods degrade significantly when boxes are inaccurate.
Approach
Simulate inaccurate boxes by scaling, translating and rotating GT boxes, then jointly learn an adaptive box-to-point pseudo labeler (similarity-based filtering and assignment of points in overlapping boxes) and a coarse-to-fine query-based segmentator that refines instances within coarse regions.
Key results
- On ScanNetV2 validation with sketchy S0 boxes, Sketchy-3DIS reaches 68.8 AP50 and 83.6 AP25, outperforming GaPro+SPFormer (70.4 AP50 is GaPro's; Sketchy-3DIS higher AP25 at 83.6 vs 79.9).
- On S3DIS Area 5 with S1 sketchy boxes, Sketchy-3DIS achieves 50.7 AP and 64.6 AP50, versus 50.5 AP and 61.2 AP50 for GaPro+ISBNet with accurate boxes.
- With the most sketchy S1 boxes on ScanNetV2 validation, Sketchy-3DIS obtains 65.8 AP50 and 83.1 AP25, far above Box2Mask (52.4 AP50) and GaPro (53.5 AP50) under sketchy supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly train the pseudo labeler and instance segmentator instead of training them disjointly (ablation) | Sketchy-3DIS · S3DIS Area 5 | AP: 45.3 -> 53.4 | Helped |
| Add box-to-point partitioning using spatial correlations (ablation) | Sketchy-3DIS pseudo labeler · ScanNetV2 validation | AP: 15.9 -> 41.8 | Helped |
| Add point-to-instance assignment for points in overlapping boxes (ablation) | Sketchy-3DIS pseudo labeler · ScanNetV2 validation | AP: 41.8 -> 45.2 | Helped |
| Add point-to-box similarity filtering for background points inside boxes (ablation) | Sketchy-3DIS pseudo labeler · ScanNetV2 validation | AP: 45.2 -> 46.0 | Helped |
| Add whole-scene and coarse-region attention (ablation) | Sketchy-3DIS Multi-level Attention Block · ScanNetV2 validation | AP: 41.3 -> 44.6 | Helped |
| Add core-region attention (ablation) | Sketchy-3DIS Multi-level Attention Block · ScanNetV2 validation | AP: 44.6 -> 44.7 | Helped |
| Add self-attention on learned instance queries (ablation) | Sketchy-3DIS Multi-level Attention Block · ScanNetV2 validation | AP: 44.7 -> 46.0 | Helped |
| Add predicted instance-box supervision (ablation) | Sketchy-3DIS instance segmentator · ScanNetV2 validation | AP: 45.0 -> 46.0 | Helped |
| Add predicted instance core-box supervision (ablation) | Sketchy-3DIS instance segmentator · ScanNetV2 validation | AP: 45.2 -> 46.0 | Helped |
| Add predicted instance-mask supervision (ablation) | Sketchy-3DIS instance segmentator · ScanNetV2 validation | AP: 45.4 -> 46.0 | Helped |
| Increase sketchy-box degree from S1 to S4 (ablation) | Sketchy-3DIS · ScanNetV2 validation | AP50: 65.8 -> 62.5 | Hurt |
| Increase sketchy-box degree from S1 to S4 (ablation) | Sketchy-3DIS · S3DIS Area 5 | AP50: 64.6 -> 50.9 | Hurt |
Surprising findings
- Sketchy-3DIS trained on sketchy boxes on S3DIS Area 5 (S0 setting, 53.4 AP) exceeds the fully supervised ISBNet baseline by 3.3 AP50 (69.1 vs 65.8).
Limitations
- Performance degrades severely when sketchy boxes are immensely inaccurate (S4 setting: 62.5 AP50 on ScanNetV2 val).
- Relies on box perturbation with small parameters (5% scale/translation, 5 degree rotation) that may not reflect real annotation noise.
- Depends on superpoint features and pretrained codebases (SPFormer/ISBNet) for training from scratch.
Experimental setup
Datasets: ScanNetV2 S3DIS
Benchmarks: ScanNetV2 S3DIS Area 5
Metrics: AP AP50 AP25
Baselines: Box2Mask WISGP GaPro CIP-WPIS BSNet UnScene3D Mask3D ISBNet SPFormer
Builds on: SPFormer ISBNet Box2Mask GaPro UnScene3D
Closest papers by meaning
- BSNet: Box-Supervised Simulation-assisted Mean Teacher for 3D Instance Segmentation (CVPR 2024 · similarity 0.91)
- Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (NeurIPS 2019 · similarity 0.90)
- Extreme Point Supervised Instance Segmentation (CVPR 2024 · similarity 0.89)
- Insightful Instance Features for 3D Instance Segmentation (CVPR 2025 · similarity 0.87)
- MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving (AAAI 2024 · similarity 0.87)
- WISH: Weakly Supervised Instance Segmentation using Heterogeneous Labels (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.