Research map / Detection and segmentation / Open-set and few-shot detection
Open-Vocabulary 3D Semantic Segmentation with Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OV3D, which uses an LVLM (LLaVA-1.5) to generate entity-level text descriptions, SEEM to segment and assign them to multi-view image pixels, and projects these to 3D points to train an open-vocabulary 3D segmentation network aligned to CLIP text features.
Problem
Open-vocabulary 3D semantic segmentation lacks fine-grained point-to-text correspondence because existing methods align 3D features to image- or region-level captions, and (point cloud, text) paired data is scarce.
Approach
Use multi-view images as a bridge with foundation models to build fine-grained point-to-entity-text pairs: an LVLM produces entity names and multiple descriptions, a segmentation/VLM model (SEEM) links pixel segments to entity texts, and 3D points inherit these texts for alignment to frozen CLIP text embeddings.
Key results
- On ScanNet annotation-free, OV3D reaches 57.3 mIoU vs 54.2 for OpenScene and 59.6 mIoU when combined with OpenScene-3D.
- On nuScenes, OV3D achieves 44.6 mIoU vs 42.1 for OpenScene-OpenSeg.
- In base-annotated B10/N9 ScanNet setting, OV3D improves harmonic mIoU over RegionPLC by +6.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LVLM to identify concrete entities from images instead of extracting nouns from captions (ablation) | OV3D · ScanNet val, annotation-free entity identification | mIoU: 55.3 -> 59.6 | Helped |
| Check extracted nouns for concreteness with WordNet (ablation) | Caption + Noun Extraction · ScanNet val, annotation-free entity identification | mIoU: 55.3 -> 57.2 | Helped |
| Add diverse entity descriptions to entity-name text embeddings (ablation) | OV3D · ScanNet val, annotation-free | mIoU: 57.6 -> 59.6 | Helped |
| Use SEEM for both class-agnostic segments and segment-to-text matching (ablation) | OV3D pixel-EntityText association · ScanNet val, annotation-free | mIoU: 55.1 -> 59.6 | Helped |
| Combine OV3D alignment objectives with OpenScene-3D (ablation) | OV3D · ScanNet val, annotation-free | mIoU: 57.3 -> 59.6; +5.7% | Helped |
| Combine OV3D alignment objectives with OpenScene-3D (ablation) | OV3D · Matterport3D test, annotation-free | mIoU: 45.8 -> 48.2; +6.2% | Helped |
| Replace OpenScene-3D with OV3D | OpenScene-3D · ScanNet200, annotation-free | mIoU: 7.3 -> 8.7; +2.5% | Helped |
| Use OV3D rather than RegionPLC | ScanNet, base-annotated B15/N4 · ScanNet val, B15/N4 | hIoU: 69.9 -> 72.4; +2.5% | Helped |
Limitations
- Relies on quality of LVLM and vision-language segmentation outputs, which can be noisy
- Requires multi-view images with depth/camera poses for point-pixel projection
Experimental setup
Datasets: ScanNet Matterport3D nuScenes ScanNet200 S3DIS
Benchmarks: ScanNet Matterport3D nuScenes ScanNet200 S3DIS
Metrics: mIoU mAcc f-mIoU f-mAcc hIoU
Baselines: OpenScene RegionPLC PLA MSeg PointCLIP MaskCLIP LSeg-3D 3DGenZ 3DTZSL
Builds on: OpenScene RegionPLC PLA SEEM CLIP
Closest papers by meaning
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.91)
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.91)
- Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.90)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.90)
- FM-OV3D: Foundation Model-Based Cross-Modal Knowledge Blending for Open-Vocabulary 3D Detection (AAAI 2024 · similarity 0.90)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.