Research map / 3D vision / Gaussian splatting
Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Segment then Splat, which partitions Gaussians into object-specific sets via SAM/SAM2 tracked multi-view masks before reconstruction, then assigns CLIP embeddings per object for open-vocabulary 3D segmentation in static and dynamic scenes.
Problem
Open-vocabulary 3D segmentation with Gaussian Splatting suffers from multi-view inconsistency, geometric/semantic ambiguity from splat-then-segment pipelines, and Gaussian-object misalignment in dynamic scenes.
Approach
Reverse the pipeline: assign Gaussians to object-specific sets (with multi-granularity IDs from tracked 2D masks) before reconstruction, supervise each object's Gaussians with a per-object mask loss, and attach one CLIP embedding per object, removing the need for a separate language field.
Key results
- On 3DOVS, the method reaches 88.53 mIoU in about 9.4 minutes, versus 82.49 mIoU for LangSplat (2D) and 47.31 for LangSplat adapted to 3D.
- On HyperNeRF dynamic scenes, mIoU is 69.48 versus 7.83 for DGD, with training time of 218 vs 1564.5 minutes.
- On LERF_OVS, mIoU is 52.10 versus 42.43 for OpenGaussian (3D), while the 2D LangSplat reaches 46.37.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Segment then Splat instead of LangSplat (2D) | 3D open-vocabulary segmentation · LERF_OVS | mIoU: 46.37 -> 52.10 | Helped |
| use Segment then Splat instead of LangSplat (2D) | 3D open-vocabulary segmentation · 3DOVS | mIoU: 82.49 -> 88.53 | Helped |
| use Segment then Splat instead of DGD | dynamic open-vocabulary segmentation · HyperNeRF | mIoU: 7.83 -> 69.48 | Helped |
| use Segment then Splat instead of DGD | dynamic open-vocabulary segmentation · Neu3D | mIoU: 1.65 -> 44.00 | Helped |
| increase supervised objects per iteration from 1 to 9 (ablation) | Segment then Splat · ramen | mIoU: 51.09 -> 56.48 | Helped |
| increase supervised objects per iteration from 1 to 9 (ablation) | Segment then Splat · waldo_kitchen | mIoU: 33.97 -> 41.59 | Helped |
| increase supervised objects per iteration from 1 to 9 (ablation) | Segment then Splat · chickchicken | mIoU: 73.11 -> 75.23 | Helped |
| increase supervised objects per iteration from 1 to 9 (ablation) | Segment then Splat · split-cookie | mIoU: 77.76 -> 81.74 | Helped |
| add partial mask filtering (ablation) | Segment then Splat · ramen | mIoU: 42.19 -> 54.38 | Helped |
| add new object detection to SAM2 tracking (ablation) | object tracking · ramen | ORR: 0.889 -> 0.934 | Helped |
| add new object detection to SAM2 tracking (ablation) | object tracking · ramen | Dup: 1 -> 2 | Hurt |
| add multi-track handling (ablation) | object tracking · ramen | Dup: 2 -> 1 | Helped |
Surprising findings
- On 3DOVS, the 2D LangSplat outperforms the method on some individual scenes, which the authors attribute to 3D retrieval revealing occluded object parts not in ground-truth masks.
- Adapting LangSplat and LEGaussians to 3D segmentation by selecting Gaussians instead of pixels performs worse than their 2D versions.
Limitations
- Relies on SAM2 tracking, which can fail in complex scenes with dense, visually similar objects.
- Cannot handle relational text queries involving multiple objects.
- Per-object CLIP embeddings lack explicit inter-object relational or contextual modeling.
Experimental setup
Datasets: 3DOVS LERF_OVS HyperNeRF Neu3D Plenoptic Video
Benchmarks: 3DOVS LERF_OVS HyperNeRF Neu3D
Metrics: mIoU optimization time PSNR LPIPS SSIM object recall rate duplicate tracking count
Baselines: LangSplat LEGaussians Gaussian Grouping OpenGaussian DGD LSeg
Builds on: 3D Gaussian Splatting Deformable 3D Gaussians Segment Anything SAM2 CLIP
Closest papers by meaning
- Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration (CVPR 2025 · similarity 0.92)
- SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting (NeurIPS 2025 · similarity 0.91)
- 3D Vision-Language Gaussian Splatting (ICLR 2025 · similarity 0.90)
- PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understanding (CVPR 2025 · similarity 0.90)
- ReferSplat: Referring Segmentation in 3D Gaussian Splatting (ICML 2025 · similarity 0.89)
- COS3D: Collaborative Open-Vocabulary 3D Segmentation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.