Research map / 3D vision / 3D generation
Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreeGS, an unsupervised 3D Gaussian Splatting framework that jointly learns semantic embeddings and view-consistent instance indices via alternating 3D clustering and 2D distillation, without any 2D/3D labels or SAM masks.
Problem
Open-vocabulary 3D scene understanding with 3DGS relies on SAM-based pseudo masks and 2D foundation model features that lack cross-view identity consistency, requiring complex preprocessing and hurting view-consistent segmentation and 3D tasks.
Approach
An IDentity-coupled Semantic Field attaches both a semantic vector and a cross-view instance index to each Gaussian; instance indices come from HDBSCAN clustering in a geometry-appearance-semantics union space, and they regularize 2D CLIP/MaskCLIP distillation, with a 2D-3D joint contrastive loss bootstrapping the two steps.
Key results
- On LERF-Mask, average 2D novel-view mIoU of 69.5 versus 72.8 for Gaussian Grouping and 63.3 for LangSplat, with no SAM supervision or preprocessing.
- On 3D-OVS, average mIoU of 77.0, matching Gaussian Grouping (77.0) and below LangSplat (84.2).
- On ScanNet scene0462, 3D detection mIoU 14.5 versus 3.3 for LangSplat; preprocessing time 0 minutes versus 141 for LangSplat.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FreeGS instead of LangSplat | 3D scene segmentation · LERF-Mask average | mIoU: 63.3 -> 69.5 | Helped |
| Use FreeGS instead of Gaussian Grouping | 3D scene segmentation · LERF-Mask average | mIoU: 72.8 -> 69.5 | Hurt |
| Use FreeGS instead of LangSplat | 3D scene segmentation · 3D-OVS average | mIoU: 84.2 -> 77.0 | Hurt |
| Use FreeGS instead of Gaussian Grouping | 3D scene segmentation · 3D-OVS average | mIoU: 77.0 -> 77.0 | No clear effect |
| Use FreeGS instead of LangSplat | 3D object detection · ScanNet scene0060 | 3D box mIoU: 5.6 -> 13.7 | Helped |
| Add 2D-3D contrastive loss (ablation) | FreeGS · LERF-Mask figurines | mIoU: 33.0 -> 62.6 | Helped |
| Add 2D-3D contrastive loss (ablation) | FreeGS · 3D-OVS bench | mIoU: 34.3 -> 86.1 | Helped |
| Add 2D-3D contrastive loss (ablation) | FreeGS · ScanNet scene0060 | 3D box mIoU: 4.9 -> 13.7 | Helped |
| Add instance-level feature distillation (ablation) | FreeGS · LERF-Mask figurines | mIoU: 33.3 -> 62.6 | Helped |
| Add instance-level feature distillation (ablation) | FreeGS · 3D-OVS bench | mIoU: 73.6 -> 86.1 | Helped |
| Add instance-level feature distillation (ablation) | FreeGS · ScanNet scene0060 | 3D box mIoU: 11.7 -> 13.7 | Helped |
| Add 3D clustering (ablation) | FreeGS · 3D-OVS bench | mIoU: 60.2 -> 86.1 | Helped |
Surprising findings
- MaskCLIP failed to distinguish 'red apple' from 'green apple' while CLIP succeeded, motivating the instance-level feature branch.
Limitations
- Semantic fields add memory and computational overhead to 3DGS.
- Gaussian misalignment and low-texture backgrounds lead to suboptimal background segmentation.
- 3D detection performance has significant room for improvement.
Experimental setup
Datasets: LERF-Mask 3D-OVS ScanNet
Benchmarks: LERF-Mask 3D-OVS ScanNet
Metrics: mIoU Recall50 Recall25
Baselines: LERF Gaussian Grouping LangSplat
Builds on: 3D Gaussian Splatting LangSplat Gaussian Grouping MaskCLIP HDBSCAN
Closest papers by meaning
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.92)
- FHGS: Feature-Homogenized Gaussian Splatting (NeurIPS 2025 · similarity 0.91)
- Efficient Decoupled Feature 3D Gaussian Splatting via Hierarchical Compression (CVPR 2025 · similarity 0.90)
- Chain of Semantics Programming in 3D Gaussian Splatting Representation for 3D Vision Grounding (CVPR 2025 · similarity 0.89)
- SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting (NeurIPS 2025 · similarity 0.89)
- ReferSplat: Referring Segmentation in 3D Gaussian Splatting (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.