Research map / Detection and segmentation / Open-set and few-shot detection
SAI3D: Segment Any Instance in 3D Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAI3D, a training-free zero-shot 3D instance segmentation method that groups scene points into superpoints, scores pairwise affinity using multi-view SAM masks, and merges them with progressive hierarchical region growing.
Problem
3D instance segmentation is limited to annotated categories; open-vocabulary CLIP-based methods cannot separate same-category instances and rely on prompts, so zero-shot fine-grained instance segmentation of 3D scenes remains hard.
Approach
Combine normal-based superpoint over-segmentation with SAM automatic masks from posed multi-view images to build a superpoint affinity graph (multi-view voting weighted by visibility), then merge superpoints via a hierarchical region-growing algorithm with progressively decreasing thresholds.
Key results
- On ScanNet++ class-agnostic segmentation, SAI3D reaches 17.1 AP / 31.1 AP50 / 49.5 AP25, above Mask3D trained on ScanNetV2 (9.9 / 17.3 / 25.8) and SAM3D (7.2 / 14.2 / 29.4).
- On ScanNetV2 class-agnostic segmentation, SAI3D achieves 30.8 AP / 50.5 AP50 versus 20.2 AP for SAM3D, while supervised Mask3D reaches 65.7 AP.
- Ablation on ScanNetV2 shows the full design improves AP from 19.5 to 30.8 (about 50% relative gain over the basic version).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SAI3D instead of SAM3D | zero-shot class-agnostic segmentation · ScanNet++ | AP: 7.2 -> 17.1 | Helped |
| use SAI3D instead of SAM3D | zero-shot class-agnostic segmentation · ScanNetV2 | AP: 20.2 -> 30.8 | Helped |
| use SAI3D instead of OpenMask3D | semantic instance segmentation · ScanNet200 | AP: 15.4 -> 12.7 | Hurt |
| use SAI3D instead of OpenMask3D | class-agnostic instance segmentation · Matterport3D | AP: 15.3 -> 21.5 | Helped |
| add 3D superpoint primitives instead of point-based region growing (ablation) | SAI3D · ScanNetV2 class-agnostic ablation | AP: 19.5 -> 24.2 | Helped |
| add multi-level merging criteria (ablation) | SAI3D with 3D superpoint primitives · ScanNetV2 class-agnostic ablation | AP: 24.2 -> 28.4 | Helped |
| add progressive growing (ablation) | SAI3D with 3D superpoint primitives · ScanNetV2 class-agnostic ablation | AP: 24.2 -> 27.6 | Helped |
| add both multi-level merging criteria and progressive growing (ablation) | SAI3D with 3D superpoint primitives · ScanNetV2 class-agnostic ablation | AP: 24.2 -> 30.8 | Helped |
Surprising findings
- Zero-shot SAI3D outperforms the supervised Mask3D (trained on ScanNetV2) on class-agnostic ScanNet++ segmentation.
- SAM3D's performance fails to improve when more than 5% of views are used, whereas SAI3D keeps improving and already exceeds SAM3D's best with 1% of views.
Limitations
- Depends heavily on the accuracy of 2D SAM masks and camera poses.
- Running time scales linearly with the number of images, making large scenes costly.
- Lags behind OpenMask3D on semantic instance segmentation in ScanNet200 because it lacks supervised mask proposals.
Experimental setup
Datasets: ScanNetV2 ScanNet200 ScanNet++ Matterport3D
Benchmarks: ScanNet++ ScanNetV2 ScanNet200 Matterport3D
Metrics: AP AP50 AP25
Baselines: Mask3D SAM3D UnScene3D OVIR-3D OpenMask3D HDBSCAN Felzenszwalb et al. Nunes et al.
Builds on: Segment Anything Model SAM3D Felzenszwalb graph-based segmentation OVSeg OpenMask3D
Closest papers by meaning
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.92)
- SA3DIP: Segment Any 3D Instance with Potential 3D Priors (NeurIPS 2024 · similarity 0.91)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.91)
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.89)
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.89)
- Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.