Research map / 3D vision / 3D generation
econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussians
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes econSG, a 3D Gaussian Splatting open-vocabulary segmentation method that mutually refines OpenSeg and SAM masks (CRR) and compresses features in a low-dimensional 3D contextual space fused from backprojected 2D features.
Problem
Open-vocabulary 3D semantic segmentation with Gaussian splatting suffers from imprecise, boundary-poor 2D VLM features, multi-view inconsistency, and high cost from high-dimensional semantic fields or per-view dimensionality reduction.
Approach
Mutually refine OpenSeg per-pixel features and SAM region masks using multi-view backprojected, confidence-filtered 3D-consistent features, then fuse 3D features first and train an autoencoder so that the 3DGS semantic field is learned in a low-dimensional latent space.
Key results
- On 3DOVS, econSG achieves 94.3 overall mIoU versus 82.3 for LangSplat and 86.8 for 3DOVS.
- On LERF localization, econSG reaches 90.5 overall accuracy versus 84.3 for LangSplat.
- On ScanNet multi-view, econSG reports 48.205 mIoU and runs at 156 FPS, versus 18.069 mIoU for Feature3DGS at 2.5 FPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use econSG instead of LangSplat | 3D semantic segmentation · 3DOVS overall | mIoU: 82.3 -> 94.3 | Helped |
| use econSG instead of SemanticGaussian | 3D open-vocabulary localization · LERF overall | localization accuracy: 85.2 -> 90.5 | Helped |
| use econSG instead of Feature3DGS | 3D semantic segmentation · Replica sparse-view | mIoU: 9.584 -> 25.513 | Helped |
| use econSG instead of Feature3DGS | 3D semantic segmentation · Replica multi-view | mIoU: 10.634 -> 33.869 | Helped |
| use econSG instead of Feature3DGS | 3D semantic segmentation · ScanNet sparse-view | mIoU: 17.552 -> 39.018 | Helped |
| use econSG instead of Feature3DGS | 3D semantic segmentation · ScanNet multi-view | mIoU: 18.069 -> 48.205 | Helped |
| increase econSG feature dimension from 6 to 16 (ablation) | econSG · 3DOVS sofa | mIoU: 91.6 -> 91.8 | No clear effect |
| increase econSG feature dimension from 6 to 32 (ablation) | econSG · 3DOVS sofa | training time (min): 29 -> 43 | Hurt |
| remove the autoencoder (ablation) | econSG · 3DOVS sofa | mIoU | Hurt |
| add CRR (ablation) | econSG · Room0 | mIoU: 13.245 -> 31.715 | Helped |
| add CRR high-confidence region selection (step a) (ablation) | econSG CRR · Replica | mIoU: 23.276 -> 33.869 | Helped |
| add SAM-based mask refinement (CRR step d) (ablation) | econSG CRR · Replica | mIoU: 27.146 -> 33.869 | Helped |
Surprising findings
- Naively removing the autoencoder with 512-dimensional features leads to out-of-memory failure during training.
- Naively over-trusting SAM masks to refine boundaries does not work well in complex scenes, per the authors' ablation.
Limitations
- Relies on Colmap depth and SfM point clouds for backprojection and initialization.
- Part-level segmentation is acknowledged as out of scope and not fully claimed.
- Requires fixed confidence thresholds (tau1=0.45, tau2=0.6) tuned per setup.
Experimental setup
Datasets: ScanNet Replica LERF 3DOVS Mip-NeRF360 LERF-Localization MVImgNet
Benchmarks: ScanNet Replica LERF 3DOVS
Metrics: mIoU mAcc FPS training time localization accuracy
Baselines: LERF 3DOVS Feature3DGS LEGaussians LangSplat SemanticGaussian LSeg
Builds on: 3D Gaussian Splatting OpenSeg Segment Anything LangSplat Gaussian Grouping
Closest papers by meaning
- GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields (CVPR 2024 · similarity 0.89)
- Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding (AAAI 2025 · similarity 0.89)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.88)
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.87)
- Chain of Semantics Programming in 3D Gaussian Splatting Representation for 3D Vision Grounding (CVPR 2025 · similarity 0.87)
- Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.