Research map / Detection and segmentation / 3D detection and driving
UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fully unsupervised class-agnostic 3D instance segmentation of indoor scans via NCut pseudo masks on geometric primitives from fused 2D/3D self-supervised features, refined by multi-round self-training.
Problem
Class-agnostic 3D instance segmentation of cluttered indoor RGB-D scans without any manual 3D annotations, where prior unsupervised clustering methods perform poorly.
Approach
Generate pseudo instance masks by applying normalized cut to geometric primitives (Felzenszwalb-style oversegmentation of mesh) with fused self-supervised color (DINO) and geometry (CSC) features, then iteratively self-train a 3D transformer on its own confident predictions, with connectivity filtering to keep masks contiguous.
Key results
- On ScanNet, UnScene3D reaches 15.9 AP, 58.5 AP@25 and 32.2 AP@50, about 3x the best clustering baseline (Felzenszwalb 5.0 AP).
- On S3DIS Area 5, UnScene3D reaches 21.4 AP versus 5.5 for the best baseline (Nunes et al.).
- Self-training improves ScanNet AP from 5.9 (S0 pseudo masks, 3D&2D) to 15.9 after 4 cycles, saturating after about 3 cycles.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UnScene3D instead of Felzenswalb clustering | UnScene3D · ScanNet | AP: 5.0 -> 15.9 | Helped |
| Use UnScene3D instead of Nunes et al. | UnScene3D · S3DIS Area 5 | AP: 5.5 -> 21.4 | Helped |
| Use both color and geometry features instead of geometry features alone for pseudo mask generation (ablation) | UnScene3D · ScanNet | AP Final: 13.3 -> 15.9 | Helped |
| Use both color and geometry features instead of color features alone for pseudo mask generation (ablation) | UnScene3D · ScanNet | AP Final: 15.7 -> 15.9 | Helped |
| Add the first self-training cycle to 3D-only pseudo masks (ablation) | UnScene3D, 3D-only · ScanNet | AP: 2 -> 9.2 | Helped |
| Add the second self-training cycle to 3D-only pseudo masks (ablation) | UnScene3D, 3D-only · ScanNet | AP: 9.2 -> 12.0 | Helped |
| Add the third self-training cycle to 3D-only pseudo masks (ablation) | UnScene3D, 3D-only · ScanNet | AP: 12.0 -> 12.8 | Helped |
| Add the fourth self-training cycle to 3D-only pseudo masks (ablation) | UnScene3D, 3D-only · ScanNet | AP: 12.8 -> 13.3 | Helped |
| Add self-training cycles to 2D and 3D pseudo masks (ablation) | UnScene3D, 2D and 3D · ScanNet | AP: 5.9 -> 15.9 | Helped |
| Add a fourth self-training cycle after the third cycle (ablation) | UnScene3D, 2D and 3D · ScanNet | AP: 15.9 -> 15.9 | No clear effect |
Surprising findings
- Projecting CutLER 2D instance masks to 3D performs very poorly (7.0 AP@25, 0.3 AP) despite plausible 2D proposals, due to view inconsistencies and occlusions.
- Fusing color and geometry features via weighted similarity averaging was more robust than direct feature fusion.
Limitations
- Graph coarsening may cause very small objects (e.g., pens, cell phones) to be missed.
- Fixed initial pseudo masks used throughout self-training could reinforce noisy predictions.
- Relies on mesh representation for graph coarsening.
Experimental setup
Datasets: ScanNet ARKitScenes S3DIS
Benchmarks: ScanNet S3DIS Area 5 ARKitScenes
Metrics: AP AP@25 AP@50
Baselines: HDBSCAN Nunes et al. Felzenszwalb CutLER Projection FreeMask
Builds on: Normalized Cut CutLER FreeSolo CSC Felzenszwalb graph segmentation
Closest papers by meaning
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.89)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.88)
- UniDet3D: Multi-dataset Indoor 3D Object Detection (AAAI 2025 · similarity 0.88)
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.87)
- FreePoint: Unsupervised Point Cloud Instance Segmentation (CVPR 2024 · similarity 0.87)
- SA3DIP: Segment Any 3D Instance with Potential 3D Priors (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.