Research map / Point clouds / Point cloud classification
OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Enhances 3D Gaussian Splatting with point-level open-vocabulary understanding by learning SAM-mask-supervised 3D-consistent instance features, discretizing them with a two-level coarse-to-fine codebook, and training-free associating CLIP features to 3D points via IoU and feature distance.
Problem
Existing 3DGS-based open-vocabulary methods work at 2D pixel level and fail at 3D point-level understanding due to weak feature expressiveness from dimension reduction and inaccurate 2D-3D feature association under alpha-blending.
Approach
Train 3D instance features directly from view-independent SAM boolean masks using intra-mask smoothing and inter-mask contrastive losses, discretize them with a position-aware coarse codebook and feature-only fine codebook, and link 3D instances to 2D SAM masks (scored by IoU times feature agreement) to inherit lossless CLIP features without training.
Key results
- On LeRF object selection in 3D, OpenGaussian reaches 38.36 mean mIoU and 51.43 mean mAcc, versus 9.66 / 12.41 for LangSplat and 16.21 / 23.82 for LEGaussians.
- On ScanNet 10-class point cloud understanding, OpenGaussian achieves 38.29 mIoU and 55.19 mAcc, versus 12.82 / 28.62 for the best baseline (LEGaussians).
- The two-level codebook raises ScanNet mIoU from 28.68 (single-level k=64) to 38.29.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OpenGaussian instead of LEGaussians | 3D text-query object selection · LeRF | mIoU: 16.21 -> 38.36 | Helped |
| use OpenGaussian instead of LEGaussians | 3D point-cloud semantic segmentation · ScanNet 19 classes | mIoU: 3.84 -> 24.73 | Helped |
| add intra-mask smoothing loss to inter-mask contrastive loss (ablation) | OpenGaussian instance feature learning · ScanNet 10 classes | mIoU: 35.24 -> 38.29 | Helped |
| use inter-mask contrastive loss instead of intra-mask smoothing loss alone (ablation) | OpenGaussian instance feature learning · ScanNet 10 classes | mIoU: 25.89 -> 35.24 | Helped |
| increase single-level codebook size from k=64 to k=320 without coordinates (ablation) | OpenGaussian codebook · ScanNet 10 classes | mIoU: 28.68 -> 14.61 | Hurt |
| increase single-level codebook size from k=64 to k=320 with coordinates (ablation) | OpenGaussian codebook · ScanNet 10 classes | mIoU: 32.04 -> 15.20 | Hurt |
| include coordinates in the coarse level of the two-level codebook (ablation) | OpenGaussian codebook · ScanNet 10 classes | mIoU: 30.27 -> 38.29 | Helped |
| use a two-level codebook instead of a single-level codebook, both with coordinates and k=64 (ablation) | OpenGaussian codebook · ScanNet 10 classes | mIoU: 32.04 -> 38.29 | Helped |
| combine IoU and feature-distance association instead of IoU alone (ablation) | OpenGaussian 2D-3D association · ScanNet 10 classes | mIoU: 35.28 -> 38.29 | Helped |
| combine IoU and feature-distance association instead of feature-distance alone (ablation) | OpenGaussian 2D-3D association · ScanNet 10 classes | mIoU: 34.01 -> 38.29 | Helped |
Surprising findings
- Simply increasing the single-level codebook size from k=64 to k=320 sharply decreases performance (28.68 to 14.61 mIoU), because spatially distant points get grouped together.
Limitations
- Gaussian geometric properties (position, opacity, scale) are fixed, so geometry and semantics may be inconsistent.
- The codebook sizes k are set empirically rather than adaptively.
- Does not address object size regression for open-vocabulary 3D detection or dynamic scenes.
Experimental setup
Datasets: LeRF-ovs ScanNetv2 Waymo
Benchmarks: LeRF-ovs ScanNetv2
Metrics: mIoU mAcc mAcc@0.25
Baselines: LangSplat LEGaussians SAGA
Builds on: 3D Gaussian Splatting LangSplat Segment Anything CLIP
Closest papers by meaning
- PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understanding (CVPR 2025 · similarity 0.91)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.90)
- SOVGaussian: Sparse-View 3D Gaussian Splatting for Open-Vocabulary Scene Understanding (AAAI 2025 · similarity 0.89)
- Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding (CVPR 2024 · similarity 0.89)
- Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting (NeurIPS 2025 · similarity 0.89)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.