Research map / Detection and segmentation / Open-set and few-shot detection
Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Open3DIS combines class-agnostic 3D instance proposals (ISBNet/Mask3D) with proposals built by aggregating 2D SAM/Grounded-SAM masks onto superpoints across RGB-D frames, then scores proposals with multi-view pointwise CLIP features for open-vocabulary 3D instance segmentation.
Problem
Open-vocabulary 3D instance segmentation struggles with small, rare, and geometrically ambiguous objects: 3D proposal networks miss them, and back-projected 2D masks give fragmented or background-contaminated 3D proposals.
Approach
A 2D-Guided-3D instance proposal module that merges superpoints using per-frame 2D mask overlap (IoU) plus 3D backbone feature similarity, then agglomerative hierarchical merging across frames, combined with 3D proposals; classification uses a 3D-aware pointwise CLIP feature accumulated over top-λ views.
Key results
- On ScanNet200, Open3DIS (2D and 3D) reaches 23.7 AP, 29.4 AP50, 32.8 AP25, about 1.5x the prior best (OpenMask3D 15.4 AP).
- 2D-only proposals reach 18.2 AP on ScanNet200 and 18.5 AP on Replica with 2D+3D proposals, beating OpenMask3D (13.1 AP) and OVIR-3D (11.1 AP) on Replica.
- On S3DIS, Open3DIS attains 26.3 novel AP50 (B8/N4) versus 13.8 for Lowis3D, more than doubling prior novel-class scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine 2D and 3D proposals rather than using 3D proposals alone | Open3DIS · ScanNet200 val | AP: 18.6 -> 23.7 | Helped |
| Use Open3DIS with 2D and 3D proposals instead of OpenMask3D | ScanNet200 val · ScanNet200 val | AP: 15.4 -> 23.7 | Helped |
| Use Open3DIS with 2D proposals instead of OpenMask3D | Open3DIS · Replica | AP: 13.1 -> 18.1 | Helped |
| Use Open3DIS instead of Lowis3D for novel-category segmentation | S3DIS Area 5 · S3DIS Area 5 | APN50: 13.8 -> 26.3 | Helped |
| Replace pointwise features with mask-wise features (ablation) | Open3DIS · ScanNet200 val | AP: 23.7 -> 22.2 | Hurt |
| Add deep feature filtering during superpoint merging (ablation) | 2D-Guided-3D proposal module · ScanNet200 val | AP: 15.9 -> 18.2 | Helped |
| Replace deep feature filtering with Euclidean distance filtering (ablation) | 2D-Guided-3D proposal module · ScanNet200 val | AP: 18.2 -> 16.0 | Hurt |
| Use superpoint merging instead of grouping 3D points solely from 2D masks (ablation) | 2D-Guided-3D proposal module · ScanNet200 val | AP: 12.0 -> 18.2 | Helped |
| Use agglomerative rather than Hungarian matching (ablation) | 2D-Guided-3D proposal module · ScanNet200 val | AP: 13.2 -> 16.9; 16.1 -> 18.2 | Helped |
| Use hierarchical rather than sequential merging (ablation) | 2D-Guided-3D proposal module · ScanNet200 val | AP: 13.2 -> 16.1; 16.9 -> 18.2 | Helped |
| Replace ISBNet with Mask3D as the 3D segmenter (ablation) | Open3DIS · ScanNet200 val | AP: 23.7 -> 23.7 | No clear effect |
| Replace Grounded-SAM with SEEM, ODISE, or Detic (ablation) | Open3DIS · ScanNet200 val | AP: 23.7 -> 21.5; 23.7 -> 21.6; 23.7 -> 22.2 | Hurt |
Surprising findings
- Using only 2D-guided proposals (no 3D network) already outperforms OpenMask3D on ScanNet200 (18.2 vs 15.4 AP).
- Mask-wise CLIP feature extraction vs pointwise: swapping in OpenScene fusion features gave only 17.5 AP, while the pointwise design reached 23.7.
Limitations
- The class-agnostic 3D proposal network and the 2D-guided proposal module operate independently; their outputs are simply combined.
- 2D masks from occluded views can be inferior, so using more views (10 or more) reduces performance.
- Performance depends on the 2D segmenter quality; SEEM, Detic, and ODISE give about 1.4 lower AP than Grounded-SAM.
Experimental setup
Datasets: ScanNet200 S3DIS Replica
Benchmarks: ScanNet200 Replica S3DIS
Metrics: AP AP50 AP25 AP head/common/tail APbase APnovel APB50 APN50
Baselines: OpenMask3D OVIR-3D OpenScene + Mask3D OpenIns3D SAM3D PLA Lowis3D ISBNet Mask3D
Builds on: OpenMask3D OVIR-3D ISBNet Grounded-SAM OpenScene
Closest papers by meaning
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.94)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.92)
- Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation (ICLR 2025 · similarity 0.91)
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.91)
- MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation (CVPR 2024 · similarity 0.91)
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.