Research map / Point clouds / Segmentation and detection
Weakly Supervised Point Cloud Semantic Segmentation via Artificial Oracle
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REAL, which uses SAM as an artificial oracle via a projection-based point-to-segment module within active-learning-style label refinement to boost sparse-label weakly supervised point cloud semantic segmentation.
Problem
Weakly supervised point cloud semantic segmentation with very sparse point labels (e.g., 1 point per object, 0.004%) is limited by the information in the initial sparse annotations, capping performance.
Approach
Project sparse labeled 3D points onto registered images, prompt SAM with them, back-project the 2D masks to get artificial 3D labels (PP2S), then iteratively query confident unlabeled points, use PP2S as a non-human oracle to label them, and conservatively update pseudo-labels against model predictions.
Key results
- On S3DIS at 0.02% labels, REAL with PTv2 reaches 65.2 mIoU, 2.9 points above CPCM (62.3).
- With only 0.004% (1pt per object) labels on S3DIS, REAL with PTv2 reaches 62.7 mIoU, above CPCM's 0.02% result.
- On ScanNetV2 test, REAL reaches 65.5 mIoU versus CPCM's 62.8 with 20 points per scene.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PP2S preprocessing (ablation) | PTv2 · S3DIS Area 5, 0.004% | mIoU: 39.7 -> 53.0 | Helped |
| Add REAL label refinement after PP2S preprocessing (ablation) | PTv2 · S3DIS Area 5, 0.004% | mIoU: 53.0 -> 62.7 | Helped |
| Add PP2S preprocessing (ablation) | PTv2 · S3DIS Area 5, 0.02% | mIoU: 55.1 -> 60.0 | Helped |
| Add REAL label refinement after PP2S preprocessing (ablation) | PTv2 · S3DIS Area 5, 0.02% | mIoU: 60.0 -> 65.2 | Helped |
| Add PP2S preprocessing (ablation) | Closer · S3DIS Area 5, 0.004% | mIoU: 38.7 -> 44.9 | Helped |
| Add REAL label refinement after PP2S preprocessing (ablation) | Closer · S3DIS Area 5, 0.004% | mIoU: 44.9 -> 57.8 | Helped |
| Use the proposed constrained query sampling strategy instead of sampling from all points (ablation) | REAL · S3DIS Area 5, 0.004% | mIoU: 59.8% -> 62.7% | Helped |
| Use REAL instead of CPCM | Weakly supervised segmentation · ScanNetV2 val, 20 points per scene | mIoU: 62.7 -> 66.3 | Helped |
| Use REAL instead of CPCM | Weakly supervised segmentation · ScanNetV2 test, 20 points per scene | mIoU: 62.8 -> 65.5 | Helped |
Surprising findings
- Using 1 point per object with REAL outperforms conventional methods that use 0.02% labels on S3DIS.
Limitations
- PP2S labels are noisy due to camera calibration errors and SAM ambiguity
- Requires image, depth, and camera projection data during training, which may not always be available
- Label updates are fixed to prior labels when model and oracle disagree, so artifacts can persist
Experimental setup
Datasets: S3DIS ScanNetV2
Benchmarks: S3DIS ScanNetV2
Metrics: mIoU
Baselines: CPCM MIL HybridCR PSD SPT Closer PTv2
Builds on: Segment Anything Model PTv2 Closer
Closest papers by meaning
- Weakly Supervised Semantic Point Cloud Segmentation: Towards 10x Fewer Labels (CVPR 2020 · similarity 0.92)
- Less Is More: Label Recommendation for Weakly Supervised Point Cloud Semantic Segmentation (AAAI 2024 · similarity 0.92)
- Multi-Path Region Mining for Weakly Supervised 3D Semantic Segmentation on Point Clouds (CVPR 2020 · similarity 0.92)
- Point Cloud Semantic Segmentation with Sparse and Inhomogeneous Annotations (AAAI 2025 · similarity 0.91)
- Multi-Modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation (AAAI 2024 · similarity 0.90)
- Distribution Guidance Network for Weakly Supervised Point Cloud Semantic Segmentation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.