Research map / Detection and segmentation / 3D detection and driving
InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstanceGaussian, which combines a Semantic-Scaffold-GS representation (shared instance features per anchor), progressive appearance-semantic joint training, and bottom-up FPS/k-means over-segmentation with graph-connectivity aggregation for category-agnostic and open-vocabulary 3D instance perception on ScanNet and LeRF.
Problem
3D Gaussian Splatting scene understanding suffers from imbalance between appearance and semantic representation, inconsistent object boundaries between appearance and semantics, and top-down instance segmentation that depends on category priors or pre-set cluster counts.
Approach
Child Gaussians decoded from a shared anchor's appearance embedding share one instance feature, training is staged (appearance, then independent, then joint) with a truncated inter-mask contrastive loss, and instances are obtained bottom-up by over-segmenting with FPS-initialized k-means and merging sub-objects by feature distance plus voxel adjacency.
Key results
- Category-agnostic instance segmentation on ScanNet: InstanceGaussian 50.27 mIoU / 80.22 mAcc vs OpenGaussian 27.32 / 52.44 and GaussianGrouping 22.55 / 30.54 (MaskClustering with depth: 54.17 / 80.95).
- Open-vocabulary 19-class ScanNet query: 40.66 mIoU / 54.01 mAcc vs OpenGaussian 24.73 / 41.54, surpassing depth-assisted MaskClustering (34.59 / 54.32).
- LeRF open-vocabulary 3D object selection: 45.30 mIoU / 58.44 mAcc vs OpenGaussian 38.36 / 51.34.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use InstanceGaussian instead of MaskClustering | ScanNet category-agnostic 3D instance segmentation · ScanNet (10 scenes) | mIoU: 54.17 -> 50.27 | Hurt |
| use InstanceGaussian instead of MaskClustering | ScanNet category-agnostic 3D instance segmentation · ScanNet (10 scenes) | mAcc.: 80.95 -> 80.22 | Hurt |
| use InstanceGaussian instead of MaskClustering | open-vocabulary point-cloud understanding · ScanNet, 15 classes | mIoU: 34.00 -> 42.51 | Helped |
| use InstanceGaussian instead of OpenGaussian | open-vocabulary 3D object selection and rendering · LeRF | mIoU: 38.36 -> 45.30 | Helped |
| use joint representation and progressive joint training (ablation) | InstanceGaussian · ScanNet | Semantic Seg. mIoU: 30.71 -> 40.66 | Helped |
| use joint representation and progressive joint training (ablation) | InstanceGaussian · ScanNet | Instance Seg. mIoU: 47.40 -> 50.27 | Helped |
| use both instance-feature similarity and voxel proximity for aggregation (ablation) | InstanceGaussian · ScanNet | Semantic Seg. mIoU: 28.98 -> 40.66 | Helped |
| use both instance-feature similarity and voxel proximity for aggregation (ablation) | InstanceGaussian · ScanNet | Instance Seg. mIoU: 43.41 -> 50.27 | Helped |
Surprising findings
- The Gaussian-based InstanceGaussian surpasses the depth-assisted MaskClustering on open-vocabulary ScanNet queries, despite trailing it on category-agnostic segmentation.
Limitations
- Longer training time and slower rendering because appearance attributes are computed through MLPs.
- Segmentation accuracy degrades when most scene images are incorrectly segmented by SAM.
Experimental setup
Datasets: ScanNet LeRF
Benchmarks: ScanNet LeRF
Metrics: mIoU mAcc@0.25
Baselines: MaskClustering GaussianGrouping OpenGaussian LangSplat LEGaussians
Builds on: 3D Gaussian Splatting Scaffold-GS OpenGaussian SAM
Closest papers by meaning
- Edge-Aware 3D Instance Segmentation Network with Intelligent Semantic Prior (CVPR 2024 · similarity 0.87)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.87)
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.87)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.87)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.87)
- Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.