Research map / Point clouds / Segmentation and detection
Point-SAM: Promptable 3D Segmentation Model for Point Clouds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Point-SAM extends SAM to point clouds with a Voronoi-tokenized transformer encoder, and distills 2D SAM knowledge via a data engine that generates part- and object-level pseudo-labels on ShapeNet for promptable 3D segmentation.
Problem
Building a native 3D promptable segmentation foundation model for point clouds, given non-unified 3D data formats, scalable architectures, and scarcity of labeled 3D masks with diverse granularity.
Approach
A SAM-style promptable point-cloud segmenter with a Voronoi-diagram tokenizer replacing KNN grouping for efficiency, trained on a mixture of labeled 3D datasets plus pseudo-labels generated by iteratively combining Point-SAM predictions with 2D SAM proposals across rendered views.
Key results
- On zero-shot point-prompted segmentation, Point-SAM reaches IoU@1 of 47.9 on PartNet-Mobility held-out categories versus 29.3 for MV-SAM and 26.4 for AGILE3D, and 49.4 IoU@1 on ScanObjectNN versus 34.8 for AGILE3D.
- The Voronoi tokenizer raises FPS on KITTI360 by 163.4% (5.2 to 13.7 shape/sec) and reduces GPU memory by 18.4% with comparable accuracy to KNN tokenization.
- Adding ShapeNet pseudo-labels from the data engine improves PartNet-Mobility IoU@1 from 38.2 (PartNet only) to 44.5, and the full mixture reaches 47.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the KNN tokenizer with the Voronoi tokenizer (ablation) | Point-SAM · PartNet-Mobility, ScanObjectNN, S3DIS, and KITTI360 | IoU@k: PartNet-Mobility IoU@1: 46.0 -> 47.9; IoU@10: 77.8 -> 78.6; ScanObjectNN IoU@1: 52.4 -> 49.4; IoU@10: 85.7 -> 86.3; S3DI | Mixed |
| use the Voronoi tokenizer instead of KNN (ablation) | Point-SAM · KITTI360 | FPS@10: 5.2 -> 13.7; +163.4% | Helped |
| use the Voronoi tokenizer instead of KNN (ablation) | Point-SAM · KITTI360 | GPU memory: 3890 -> 3172 MB; -18.4% | Helped |
| add ScanNet to PartNet training data (ablation) | Point-SAM · PartNet-Mobility | IoU@1: 38.2 -> 39.7 | Helped |
| add ShapeNet pseudo labels to PartNet training data (ablation) | Point-SAM · PartNet-Mobility | IoU@1: 38.2 -> 44.5 | Helped |
| add ShapeNet pseudo labels and ScanNet to PartNet training data (ablation) | Point-SAM · PartNet-Mobility | IoU@1: 38.2 -> 45.4 | Helped |
| train on ScanNet only rather than the full training mixture (ablation) | Point-SAM · S3DIS | IoU@1: 47.6 -> 38.8 | Hurt |
| train on ScanNet only rather than the full training mixture (ablation) | Point-SAM · KITTI360 | IoU@1: 49.4 -> 44.0 | Hurt |
| generate ShapeNet pseudo labels with Point-SAM alone, without SAM (ablation) | Point-SAM data engine · PartNet-Mobility | IoU@1: 38.2 -> 44.5 | Helped |
| generate ShapeNet pseudo labels with Point-SAM alone, without SAM (ablation) | Point-SAM data engine · PartNet-Mobility | IoU@3: 65.2 -> 56.7 | Hurt |
| increase patch size from 64 to 256 while keeping 512 patches (ablation) | Point-SAM · S3DIS | IoU@10: 76.7 -> 80.2 | Helped |
| increase patch count from 512 to 2048 while keeping patch size 64 (ablation) | Point-SAM · S3DIS | IoU@10: 76.7 -> 85.8 | Helped |
Surprising findings
- Point-SAM trained only on ScanNet (Point-SAM*) still outperforms AGILE3D on OOD datasets KITTI360 and PartNet-Mobility, though it is worse on in-domain S3DIS.
- The Voronoi tokenizer achieves comparable accuracy to the KNN tokenizer on several benchmarks while being more efficient; the KNN variant is slightly better on some datasets.
Limitations
- Reliance on pseudo-labels from SAM inherits 2D label quality and may miss 3D nuances.
- 3D datasets remain far less diverse and smaller than 2D data, limiting generalization.
- Computational demands of large-scale 3D processing and lack of efficient 3D-specific operators remain hurdles.
Experimental setup
Datasets: PartNet PartNet-Mobility ScanNet200 ScanNet-Block Fusion360 ShapeNet ScanObjectNN S3DIS KITTI-360 Replica ShapeNetPart
Benchmarks: PartNet-Mobility ScanObjectNN S3DIS KITTI360 Replica ShapeNetPart StorageFurniture
Metrics: IoU@k average recall (AR) FPS GPU memory
Baselines: MV-SAM AGILE3D InterObject3D++ OpenMask3D Uni3D PointBERT
Builds on: Segment Anything Model (SAM) Point Transformer AGILE3D PointBERT
Closest papers by meaning
- ODIN: A Single Model for 2D and 3D Segmentation (CVPR 2024 · similarity 0.88)
- OneFormer3D: One Transformer for Unified Point Cloud Segmentation (CVPR 2024 · similarity 0.88)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.88)
- FreePoint: Unsupervised Point Cloud Instance Segmentation (CVPR 2024 · similarity 0.88)
- LogoSP: Local-global Grouping of Superpoints for Unsupervised Semantic Segmentation of 3D Point Clouds (CVPR 2025 · similarity 0.87)
- X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.