Research map / 3D vision / 3D generation
SA3DIP: Segment Any 3D Instance with Potential 3D Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free open-world 3D class-agnostic instance segmentation that merges superpoints using geometric+color priors, SAM 2D mask affinities, and a 3D-detector-based refinement, plus a corrected ScanNetV2-INS validation benchmark.
Problem
Zero-shot 3D instance segmentation from 2D foundation models suffers from under-segmentation of geometrically similar instances (from normal-only superpoints) and over-segmentation from SAM's part-level masks, plus low-quality ScanNetV2 annotations distort evaluation.
Approach
Build superpoint primitives from both normal and color cues, merge them with SAM-derived multi-view affinities via affinity- and distance-aware region growing, then rectify over-segmentation using bounding boxes from a 3D detector; also re-annotate ScanNetV2 validation scenes point-wise.
Key results
- On ScanNetV2, SA3DIP reaches 41.6 mAP, 64.6 AP50, 81.3 AP25, outperforming SAMPro3D (33.7/56.2/75.3).
- On ScanNetV2-INS, SA3DIP reaches 36.1 mAP, improving over SAMPro3D's 32.5 mAP by 3.6 points.
- On ScanNet++, SA3DIP reaches 21.4 mAP versus 18.9 for SAMPro3D, while the best-performing prior 3D-space-prior ablation on Matterport3D reaches 20.6 AP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SA3DIP instead of SAMPro3D | 3D instance segmentation · ScanNetV2 | mAP: +7.9% mAP | Helped |
| Use SA3DIP instead of SAMPro3D | 3D instance segmentation · ScanNetV2-INS | mAP: 3.6% mAP | Helped |
| Use SA3DIP instead of SAMPro3D | 3D instance segmentation · ScanNet++ | mAP: 2.5% mAP | Helped |
| Add 3D space prior to geometry weight 0.96 and texture weight 0.04 (ablation) | SA3DIP · ScanNetV2 | mAP: 29.3 -> 41.6 | Helped |
| Add 3D space prior to geometry weight 0.96 and texture weight 0.04 (ablation) | SA3DIP · ScanNetV2-INS | mAP: 27.4 -> 36.1 | Helped |
| Use geometry weight 0.96 and texture weight 0.04 instead of geometry-only primitives (ablation) | SA3DIP with 3D space prior · ScanNetV2 | mAP: 40.8 -> 41.6 | Helped |
| Use texture-only primitives instead of geometry-only primitives (ablation) | SA3DIP with 3D space prior · ScanNetV2 | mAP: 40.8 -> 12.7 | Hurt |
| Use geometry weight 0.4 and texture weight 0.6 instead of geometry-only primitives (ablation) | SA3DIP with 3D space prior · ScanNetV2 | mAP: 40.8 -> 39.1 | Hurt |
| Use geometry weight 0.96 and texture weight 0.04 instead of geometry-only primitives (ablation) | SA3DIP without 3D space prior · ScanNetV2 | mAP: 30.8 -> 29.3 | Hurt |
| Use geometry weight 0.96 and texture weight 0.04 instead of geometry-only primitives (ablation) | SA3DIP with 3D space prior · ScanNetV2-INS | mAP: 35.9 -> 36.1 | Helped |
| Use texture-only primitives instead of geometry-only primitives (ablation) | SA3DIP without 3D space prior · ScanNetV2 | mAP: 30.8 -> 10.4 | Hurt |
| Add 3D space prior (ablation) | SA3DIP #2 with geometry weight 0.9 and texture weight 0.1 · Replica | AP: 20.8 -> 22.6 | Helped |
Surprising findings
- Using only complementary geometric-plus-color primitives without the 3D detector prior slightly lowers AP versus the SAI3D-style setting, which the authors attribute to AP favoring under-segmentation.
- Texture cues alone (w_c=1) perform very poorly (10.4 mAP on ScanNetV2), so color must be weighted lightly.
Limitations
- Superpoints built only from 3D priors can yield an overwhelming number of superpoints, complicating merging.
- Affinity matrix still depends heavily on the accuracy of 2D foundation segmenters such as SAM.
- ScanNetV2-INS covers only the 312 validation scenes and targets no-training methods.
Experimental setup
Datasets: ScanNetV2 ScanNet++ Matterport3D Replica
Benchmarks: ScanNetV2 ScanNetV2-INS ScanNet++ Matterport3D Replica
Metrics: mAP AP50 AP25
Baselines: Mask3D Felzenszwalb SAM3D SAM-graph SAI3D SAMPro3D OpenMask3D OVIR-3D
Builds on: SAM3D SAI3D SAMPro3D Segment Anything Felzenszwalb graph-based segmentation
Closest papers by meaning
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.91)
- SAM2Object: Consolidating View Consistency via SAM2 for Zero-Shot 3D Instance Segmentation (CVPR 2025 · similarity 0.89)
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.88)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.87)
- Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation (ICLR 2025 · similarity 0.87)
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.