Research map / 3D vision / Gaussian splatting
3D Vision-Language Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a 3D vision-language Gaussian splatting model with a cross-modal rasterizer (self-attention modality fusion plus a learnable smoothed semantic indicator) and camera-view blending regularization, reaching state-of-the-art open-vocabulary segmentation on LERF, 3D-OVS and Mip-NeRF 360.
Problem
Multi-modal 3D scene understanding with Gaussian splatting naively embeds language features into color-based rasterization, which fails on translucent or reflective objects and overfits the color modality, hurting open-vocabulary semantic segmentation.
Approach
Fuse color and language features with self-attention before rasterization, rasterize language with a separate learnable semantic indicator instead of color opacity, and regularize with camera-view blending that interpolates camera poses (Slerp/Lerp) and ground-truth language maps weighted by SSIM.
Key results
- On LERF, mIoU averages 62.0 versus 51.4 for the second-best LangSplat (reported as a 10.6-point gain).
- Localization accuracy on LERF averages 96.0% versus 91.5% for FMGS.
- On 3D-OVS (five scenes), mIoU averages 97.1 versus 93.4 for LangSplat.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed method instead of LangSplat | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 51.4 -> 62.0 | Helped |
| Use the proposed method instead of LangSplat | 3D Vision-Language Gaussian Splatting · 3D-OVS | mIoU (avg.): 93.4 -> 97.1 | Helped |
| Use the proposed method instead of GOI | 3D Vision-Language Gaussian Splatting · Mip-NeRF 360 | mIoU (avg.): 86.5 -> 89.8 | Helped |
| Use self-attention modality fusion instead of no fusion, single-layer MLP, or cross-attention (ablation) | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 58.3, 56.8, 58.7 -> 62.0 | Helped |
| Learn a smoothed semantic indicator instead of fixing it to 0.5, fixing it to 1.0, or using color opacity (ablation) | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 56.1, 53.4, 58.1 -> 62.0 | Helped |
| Sample interpolation ratios using Beta(0.2, 0.2) instead of the reported fixed or alternate distributions (ablation) | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 60.1, 57.4, 55.8, 59.0, 58.4 -> 62.0 | Helped |
| Add SSIM weighting to translation-only camera blending (ablation) | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 56.8 -> 58.2 | Helped |
| Add SSIM weighting to rotation-only camera blending (ablation) | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 56.6 -> 58.1 | Helped |
| Add rotation and translation blending and SSIM weighting rather than omitting camera-view blending (ablation) | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 55.8 -> 62.0 | Helped |
| Use offline 3DGS-rendered images labeled by SAM/CLIP instead of online view blending (ablation) | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 55.3 -> 62.0 | Hurt |
| Keep position and covariance shared while using separate opacity/semantic indicator parameters (ablation) | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 58.1 -> 62.0 | Helped |
| Use separate covariance matrices for the two modalities (ablation) | 3D Vision-Language Gaussian Splatting · LERF | mIoU (avg.): 62.0 -> 49.2 | Hurt |
Surprising findings
- Learnable semantic indicator beats reusing color opacity, while fixing it to 0.5 or 1.0 hurts performance.
Limitations
- Evaluated only on static scenes; dynamic scenes are left to future work.
- Relies on off-the-shelf SAM and CLIP features as supervision, which are not improved by the method.
- Some baselines (FMGS) lack open code and mIoU results on LERF are missing for it.
Experimental setup
Datasets: LERF 3D-OVS Mip-NeRF 360
Benchmarks: LERF 3D-OVS Mip-NeRF 360
Metrics: mIoU localization accuracy PSNR SSIM training time FPS
Baselines: Feature-3DGS LEGaussians LangSplat GS-Grouping GOI FMGS
Builds on: 3D Gaussian splatting LangSplat CLIP Mix-up augmentation
Closest papers by meaning
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.91)
- Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration (CVPR 2025 · similarity 0.91)
- Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting (NeurIPS 2025 · similarity 0.90)
- SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting (NeurIPS 2025 · similarity 0.90)
- ReferSplat: Referring Segmentation in 3D Gaussian Splatting (ICML 2025 · similarity 0.89)
- Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.