Research map / Vision-language and multimodal / Human and embodied modeling
XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes XMask3D, which uses the denoising UNet of a pre-trained diffusion model as a geometry-conditioned 2D mask generator to align pooled 3D mask embeddings with the CLIP text space for open-vocabulary 3D semantic segmentation.
Problem
Open vocabulary 3D semantic segmentation relies on coarse global, point-wise, or patch-wise alignment or distillation to vision-language spaces, which struggles with fine-grained segmentation boundaries and novel categories.
Approach
Use the frozen denoising UNet of a text-to-image diffusion model (ODISE-style) as a 2D mask generator conditioned on global 3D point-cloud features, back-project the generated masks to 3D to pool mask embeddings for mask-level contrastive alignment with CLIP, and fuse 2D and 3D mask features.
Key results
- On ScanNet B15/N4, XMask3D reaches 70.0 hIoU (69.8 base, 70.2 novel), versus 65.3 hIoU for PLA and 72.4 hIoU for OV3D.
- Adding the mask-level regularization raises 3D novel-category mIoU by 11.7 on the ScanNet B12/N7 ablation, and fusion improves the fused output by 1.5.
- The fused output outperforms both the 2D and 3D intermediate branches on both base and novel splits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use XMask3D instead of OV3D | ScanNet B15/N4 · ScanNet B15/N4 | Novel mIoU: 70.2 vs 74.7 | Hurt |
| use XMask3D instead of OV3D | ScanNet B12/N7 · ScanNet B12/N7 | Novel mIoU: 55.1 vs 63.7 | Hurt |
| use XMask3D instead of OV3D | ScanNet B10/N9 · ScanNet B10/N9 | Novel mIoU: 43.8 vs 55.6 | Hurt |
| use XMask3D instead of OpenScene | ScanNet200 B170/N30 · ScanNet200 B170/N30 | Novel mIoU: 13.3 vs 10.4 | Helped |
| use XMask3D instead of OpenScene | ScanNet200 B150/N50 · ScanNet200 B150/N50 | Novel mIoU: 11.4 vs 11.2 | Helped |
| add mask-level regularization loss (ablation) | XMask3D 3D branch · ScanNet B12/N7 | Novel mIoU: +11.7 | Helped |
| add mask-level regularization loss (ablation) | XMask3D fusion output · ScanNet B12/N7 | performance: +1.5 | Helped |
| condition the diffusion model on global 3D features rather than a vanilla text or implicit 2D condition (ablation) | XMask3D mask generator · ScanNet B12/N7 | Novel category segmentation | Helped |
| fuse 2D mask features with 3D features (ablation) | XMask3D · ScanNet B12/N7 | base and novel mIoU | Helped |
Surprising findings
- OV3D reports higher hIoU than XMask3D on ScanNet B15/N4 (72.4 vs 70.0) and B12/N7 (68.5 vs 61.7), so XMask3D is not best on all benchmarks.
Limitations
- The 2D diffusion-based branch has many parameters and slow latency (computational cost).
- Evaluated only on semantic segmentation; instance and panoptic tasks left for future work.
- Failure cases show over-reliance on geometry and confusions between similar categories (e.g., shower curtain vs curtain).
Experimental setup
Datasets: ScanNet ScanNet200 S3DIS
Benchmarks: ScanNet B15/N4 ScanNet B12/N7 ScanNet B10/N9 ScanNet200 B170/N30 ScanNet200 B150/N50 S3DIS B8/N4 S3DIS B6/N6
Metrics: mIoU base mIoU novel hIoU
Baselines: LSeg-3D 3DGenZ 3DTZSL PLA OpenScene OV3D
Builds on: ODISE PLA Mask2Former MinkUNet MaskCLIP
Closest papers by meaning
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.91)
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.90)
- Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.88)
- Open-Vocabulary 3D Semantic Segmentation with Foundation Models (CVPR 2024 · similarity 0.88)
- MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation (CVPR 2024 · similarity 0.88)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.