Research map / 3D vision / 3D generation
RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RegionPLC, a regional point-language contrastive framework that fuses region-level captions from multiple 2D VL models (SFusion) and trains a 3D backbone with a region-aware point-discriminative contrastive loss for open-world 3D segmentation.
Problem
Open-world 3D scene understanding with only sparse or coarse language supervision and no human 3D annotations for novel categories; existing 3D-language methods use coarse captions and feature distillation is costly.
Approach
Fuse region-level 3D-language pairs from multiple 2D foundation models (detector, captioner, dense captioners) using 3D overlap-based supplementary selection (SFusion), and replace CLIP-style pooled contrastive loss with a point-wise discriminative loss normalized by region size.
Key results
- Outperforms PLA on unseen-category mIoU by 8.3%-21.8% on ScanNet and nuScenes semantic segmentation, averaging 17.2% gains.
- Annotation-free ScanNet semantic segmentation reaches 56.9 mIoU (SparseUNet16) vs 57.2 for OpenScene-3D, at about 17% of OpenScene's training cost and 5% of storage.
- Combined with OpenScene, reaches 63.6 mIoU on ScanNet annotation-free setting, about 4% gain over OpenScene alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RegionPLC instead of PLA | open-world semantic segmentation · ScanNet B15/N4 | mIoUN: 62.4 -> 70.7 | Helped |
| use RegionPLC instead of PLA | open-world semantic segmentation · ScanNet B12/N7 | mIoUN: 45.9 -> 66.6 | Helped |
| use RegionPLC instead of PLA | open-world semantic segmentation · ScanNet B10/N9 | mIoUN: 40.8 -> 55.6 | Helped |
| use RegionPLC instead of PLA | open-world semantic segmentation · nuScenes B12/N3 | mIoUN: 35.4 -> 56.0 | Helped |
| use RegionPLC instead of PLA | open-world semantic segmentation · nuScenes B10/N5 | mIoUN: 14.5 -> 36.3 | Helped |
| use RegionPLC instead of PLA | open-world semantic segmentation · ScanNet200 B170/N30 | mIoUN: 7.8 -> 13.9 | Helped |
| use RegionPLC instead of PLA | open-world semantic segmentation · ScanNet200 B150/N50 | mIoUN: 6.6 -> 10.8 | Helped |
| use RegionPLC instead of PLA | open-world instance segmentation · ScanNet B13/N4 | mAPN50: 52.9 -> 57.2 | Helped |
| use RegionPLC instead of PLA | open-world instance segmentation · ScanNet B10/N7 | mAPN50: 21.9 -> 32.5 | Helped |
| use RegionPLC instead of PLA | open-world instance segmentation · ScanNet B8/N9 | mAPN50: 25.1 -> 37.4 | Helped |
| add regional captions instead of using view- and entity-level language supervision (ablation) | RegionPLC · ScanNet B0/N17 | mIoU†: 21.7 -> 50.6 | Helped |
| add point-discriminative contrastive loss to regional language supervision (ablation) | RegionPLC · ScanNet B0/N17 | mIoU†: 21.7 -> 50.6 | Helped |
Surprising findings
- Common data-level and loss-level combinations of multiple caption sources often give inconsistent or only incremental gains, motivating SFusion.
Limitations
- Relies on quality of 2D foundation model captions, which may carry biases and conflicts.
- Annotation-free gains depend on dense captioners; detector-based captions have limited vocabulary.
Experimental setup
Datasets: ScanNet ScanNet200 nuScenes
Benchmarks: ScanNet ScanNet200 nuScenes
Metrics: hIoU mIoU mAP50 mIoU (foreground) mAcc
Baselines: PLA OpenScene OVSeg-3D 3DGenZ 3DTZSL MaskCLIP
Builds on: CLIP PLA OpenScene SoftGroup Detic
Closest papers by meaning
- Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding (AAAI 2024 · similarity 0.91)
- Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.91)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.89)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.89)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.89)
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.