Research map / 3D vision / 3D generation
MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaskClustering, a training-free method that builds a global 2D mask graph using a view consensus rate and iteratively clusters masks into 3D instances for open-vocabulary 3D instance segmentation.
Problem
Open-vocabulary 3D instance segmentation is limited by scarce annotated 3D data; existing 2D-to-3D merging methods rely on local, frame-pair metrics that lack global consistency across views.
Approach
A view consensus rate measures the fraction of co-observing frames where some other 2D mask contains both masks; using it as an edge weight in a global mask graph, masks are iteratively clustered (connected components with observer-count thresholds) into 3D instances.
Key results
- On ScanNet200, the method reaches 12.0 AP, 23.3 AP50 and 30.1 AP25 for semantic segmentation in a fully zero-shot setting, versus 9.3/18.7/25.0 for OVIR-3D.
- On ScanNet++ class-agnostic segmentation it achieves 24.6 AP versus 19.4 for OVIR-3D; on MatterPort3D semantic AP it reaches 9.2 versus 6.3 for OVIR-3D.
- Ablation shows the iterative clustering and under-segment filtering each add gains over the connected-component baseline (10.0 to 12.0 AP on ScanNet200).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaskClustering instead of OVIR-3D | zero-shot 3D instance segmentation · ScanNet++ semantic | AP: 3.6 -> 7.8 | Helped |
| Use MaskClustering instead of Mask3D | zero-shot 3D instance segmentation · ScanNet++ class-agnostic | AP: 22.8 -> 24.6 | Helped |
| Use MaskClustering instead of OVIR-3D | zero-shot 3D instance segmentation · MatterPort3D semantic | AP: 6.3 -> 9.2 | Helped |
| Use MaskClustering instead of OVIR-3D | zero-shot 3D instance segmentation · MatterPort3D class-agnostic | AP: 5.9 -> 8.3 | Helped |
| Use MaskClustering instead of OVIR-3D | zero-shot 3D instance segmentation · ScanNet200 semantic | AP: 9.3 -> 12.0 | Helped |
| Use MaskClustering instead of OVIR-3D | zero-shot 3D instance segmentation · ScanNet200 class-agnostic | AP: 14.4 -> 19.7 | Helped |
| Add under-segment mask filtering (ablation) | view-consensus connected-component baseline · ScanNet200 | AP: 10.0 -> 11.0 | Helped |
| Add iterative clustering (ablation) | view-consensus connected-component baseline · ScanNet200 | AP: 10.0 -> 11.7 | Helped |
| Use both under-segment mask filtering and iterative clustering (ablation) | view-consensus mask clustering · ScanNet200 | AP: 10.0 -> 12.0 | Helped |
| Replace iterative clustering with connected-component clustering (ablation) | MaskClustering · ScanNet200 | AP: 12.0 -> 11.0 | Hurt |
| Replace iterative clustering with clique clustering (ablation) | MaskClustering · ScanNet200 | AP: 12.0 -> 11.3 | Hurt |
| Replace iterative clustering with Quasi-Clique (HCS) (ablation) | MaskClustering · ScanNet200 | AP: 11.9 vs 12.0 | No clear effect |
Surprising findings
- OpenMask3D's mask predictor (Mask3D) is sensitive to point distribution: class-agnostic AP on ScanNet++ drops to 13.6 with raw point clouds but rises to 22.7 with uniform sampling.
Limitations
- Assumes near-perfect 2D segmentation and 2D-3D correspondence.
- Only produces object-level masks, not multi-level part/object/cluster masks.
Experimental setup
Datasets: ScanNet++ ScanNet200 MatterPort3D
Benchmarks: ScanNet++ ScanNet200 MatterPort3D
Metrics: AP AP50 AP25
Baselines: Mask3D OpenMask3D OVIR-3D
Builds on: OVIR-3D OpenMask3D CropFormer CLIP
Closest papers by meaning
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.91)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.91)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.90)
- XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation (NeurIPS 2024 · similarity 0.88)
- OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging (CVPR 2025 · similarity 0.88)
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.