Research map / 3D vision / 3D generation
SAM2Object: Consolidating View Consistency via SAM2 for Zero-Shot 3D Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAM2Object, a zero-shot 3D instance segmentation method that uses SAM2 segmentation and tracking across frames plus distance/visibility/purity-weighted mask consolidation to lift 2D masks to superpoint graphs.
Problem
Zero-shot 3D instance segmentation via 2D-to-3D lifting of vision foundation model masks suffers from view inconsistency across frames and long-distance projection errors, producing fragmented 3D segmentation.
Approach
Use SAM2's image predictor on keyframes and video predictor for bidirectional tracking to make 2D masks consistent across views, then weight superpoint-mask affinities by visibility, distance from camera, and mask purity before iterative graph clustering.
Key results
- Class-agnostic on ScanNetV2: 34.0 mAP, 52.7 AP50, 70.3 AP25, outperforming SAI3D (30.8 mAP) on mAP and AP50.
- Class-agnostic on ScanNet++: 20.2 mAP and 34.1 AP50 vs SAI3D 17.1 and 31.1.
- Semantic instance segmentation on ScanNet200: 13.3 mAP and 19.0 AP50, above SAI3D (12.7, 18.8) but AP25 lower (23.8 vs 24.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAM2Object instead of SAI3D | Zero-shot 3D instance segmentation · ScanNetV2 | mAP: 30.8 -> 34.0 | Helped |
| Use SAM2Object instead of SAI3D | Zero-shot 3D instance segmentation · ScanNetV2 | AP50: 50.5 -> 52.7 | Helped |
| Use SAM2Object instead of SAI3D | Zero-shot 3D instance segmentation · ScanNetV2 | AP25: 70.6 -> 70.3 | Hurt |
| Use SAM2Object instead of SAI3D | Zero-shot 3D instance segmentation · ScanNet++ | mAP: 17.1 -> 20.2 | Helped |
| Use SAM2Object instead of SAI3D | Zero-shot 3D instance segmentation · ScanNet++ | AP50: 31.1 -> 34.1 | Helped |
| Use SAM2Object instead of SAI3D | Zero-shot 3D instance segmentation · ScanNet++ | AP25: 49.5 -> 48.7 | Hurt |
| Use SAM2Object instead of SAI3D | Zero-shot semantic instance segmentation · ScanNet200 | mAP: 12.7 -> 13.3 | Helped |
| Use SAM2Object instead of SAI3D | Zero-shot semantic instance segmentation · ScanNet200 | AP50: 18.8 -> 19.0 | Helped |
| Use SAM2Object instead of SAI3D | Zero-shot semantic instance segmentation · ScanNet200 | AP25: 24.1 -> 23.8 | Hurt |
| Add SAM2 video object tracking (ablation) | Frame-by-frame SAM2 lifting baseline · ScanNetV2 | mAP: 28.2 -> 29.3 | Helped |
| Add mask consolidation (ablation) | Frame-by-frame SAM2 lifting baseline · ScanNetV2 | mAP: 28.2 -> 32.1 | Helped |
| Combine SAM2 video tracking and mask consolidation (ablation) | Frame-by-frame SAM2 lifting baseline · ScanNetV2 | mAP: 28.2 -> 34.0 | Helped |
Surprising findings
- Replacing SAI3D's segmentation model with SAM2 yields lower mAP than SAI3D's tuned models (28.23% on ScanNetV2, 16.02% on ScanNet++), showing per-dataset model choice matters.
Limitations
- Requires continuous, dense multi-view images for tracking.
- Vision foundation models operate on 2D data and lack 3D scene understanding.
- Less effective than methods integrating 3D priors with 2D semantics for view consistency; lower AP25 than SAI3D.
Experimental setup
Datasets: ScanNetV2 ScanNet++ ScanNet200 S3DIS
Benchmarks: ScanNetV2 ScanNet++ ScanNet200
Metrics: mAP AP50 AP25
Baselines: Mask3D SAI3D SAM3D SAMPro3D SAM-graph Segment3D OpenMask3D OVIR-3D UnScene3D Part2Object HDBSCAN UCIS
Builds on: SAM2 SAI3D Segment Anything Model Mask3D
Closest papers by meaning
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.90)
- SA3DIP: Segment Any 3D Instance with Potential 3D Priors (NeurIPS 2024 · similarity 0.89)
- OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging (CVPR 2025 · similarity 0.88)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.88)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.87)
- Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.