Research map / Point clouds / Segmentation and detection
Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MPEC, a masked point-entity contrastive pretraining framework that aligns 3D entity features across masked views and with CLIP-embedded language, achieving SOTA open-vocabulary 3D semantic segmentation on ScanNet and strong fine-tuning results.
Problem
Open-vocabulary 3D scene understanding needs scene representations that are both language-aligned and entity-specific with accurate spatial/geometric information, which existing 2D-distillation and point-only contrastive pretraining methods lack.
Approach
Combines cross-view masked point-to-entity contrastive learning (entity masks from an off-the-shelf 3D segmenter, background handled as extra negatives) with entity-to-language contrast using CLIP text embeddings and a binary cross-entropy text-entity loss to handle multiple descriptions per entity.
Key results
- Achieves 66.0% f-mIoU and 81.3% f-mAcc on ScanNet open-vocabulary semantic segmentation, outperforming OV3D (64.0 / 76.3).
- Zero-shot on SceneVerse-val reaches 45.0 f-mIoU vs 41.3 for OpenScene; ScanNet200 open-vocab reaches 10.8 f-mIoU vs 9.1 for RegionPLC.
- Fine-tuned on ScanNet200 semantic segmentation reaches 31.8 mIoU vs 30.0 for GroupContrast; with PQ3D head SQA3D rises from 47.1 to 47.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the PQ3D 3D encoder with MPEC | PQ3D · ScanRefer, Sr3D, and Multi3DRefer | grounding accuracy or F1@0.5: 51.2 -> 51.8; 79.7 -> 80.0; 50.1 -> 50.8 | Helped |
| replace the PQ3D 3D encoder with MPEC | PQ3D · Nr3D | Acc: 66.7 -> 66.7 | No clear effect |
| replace the PQ3D 3D encoder with MPEC | PQ3D · Scan2Cap | CIDEr@0.5: 80.3 -> 80.2 | Hurt |
| replace the PQ3D 3D encoder with MPEC | PQ3D · SQA3D | AccQA: 47.1 -> 47.5 | Helped |
| use cross-view augmentation and point-to-entity loss (ablation) | MPEC · ScanNet OV-SemSeg | f-mIoU: 63.6 -> 64.6 | Helped |
| use cross-view augmentation and point-to-entity loss (ablation) | MPEC · ScanNet OV-SemSeg | f-mAcc: 79.2 -> 79.5 | Helped |
| add point-to-entity loss without cross-view augmentation (ablation) | MPEC · ScanNet OV-SemSeg | f-mIoU: 63.6 -> 62.2 | Hurt |
| add captions to referral text (ablation) | MPEC · ScanNet OV-SemSeg | f-mIoU: 61.7 -> 64.6 | Helped |
| add referrals to caption text (ablation) | MPEC · ScanNet OV-SemSeg | f-mIoU: 57.6 -> 64.6 | Helped |
| add MultiScan training data to ScanNet training data (ablation) | MPEC · ScanNet OV-SemSeg | f-mIoU: 56.4 -> 57.7 | Helped |
| add RScan training data to ScanNet training data (ablation) | MPEC · ScanNet OV-SemSeg | f-mIoU: 56.4 -> 59.9 | Helped |
| add HM3D training data to ScanNet training data (ablation) | MPEC · ScanNet OV-SemSeg | f-mIoU: 56.4 -> 63.3 | Helped |
Surprising findings
- Directly combining point-to-entity loss without cross-mask augmentation hurts performance (f-mIoU drops from 63.6 to 62.2), suggesting over-emphasis on within-view entity uniqueness erodes shared attributes.
Limitations
- Performance degrades on visual grounding with similar distractors; the fixed CLIP text encoder struggles with long, detailed descriptions.
- Relies on off-the-shelf 3D instance segmentation and 2D captioning models to generate entity masks and text, which adds dependency on those generators.
Experimental setup
Datasets: SceneVerse ScanNet MultiScan 3RScan HM3D ARKitScenes Structured3D ProcTHOR ScanNet200 Matterport3D ScanRefer Nr3D
Benchmarks: ScanNet SceneVerse-val Matterport3D ScanNet200 ScanRefer Nr3D Sr3D Multi3dRefer Scan2Cap SQA3D
Metrics: f-mIoU f-mAcc mIoU mAP@0.5 Acc@0.5 F1@0.5 CIDEr@0.5 AccQA
Baselines: OpenScene OV3D RegionPLC PLA MaskCLIP CLIP2 Uni3D PC CSC MSC GroupContrast 3D-VisTA
Builds on: OpenScene GroupContrast MSC SceneVerse PQ3D
Closest papers by meaning
- RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding (CVPR 2024 · similarity 0.91)
- Open-Vocabulary 3D Semantic Segmentation with Foundation Models (CVPR 2024 · similarity 0.90)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.90)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.90)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.90)
- Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.