Research map / 3D vision / Gaussian splatting
LangSplat: 3D Language Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 3D language field by attaching SAM-hierarchy CLIP features to 3D Gaussians compressed via a scene-specific autoencoder, enabling fast open-vocabulary 3D querying.
Problem
Existing 3D language fields built on NeRF with CLIP features are slow to render and produce imprecise, vague object boundaries, and need multi-scale patch querying at inference.
Approach
Replace NeRF with 3D Gaussian Splatting carrying language embeddings, supervise with CLIP features from SAM masks at three semantic levels (subpart/part/whole) instead of multi-scale patches or DINO regularization, and learn in a low-dimensional scene-specific latent space via an autoencoder.
Key results
- LangSplat reaches 84.3% overall localization accuracy on the LERF dataset versus 73.6% for LERF.
- LangSplat achieves 93.4% overall mIoU on 3D-OVS, versus 86.8% for the prior 3D-OVS method.
- LangSplat is 199x faster than LERF per text query at 1440x1080 resolution on an RTX-3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LangSplat instead of LERF | 3D object localization · LERF dataset (overall) | localization accuracy (%): 73.6 -> 84.3 | Helped |
| Use LangSplat instead of LERF | 3D semantic segmentation · LERF dataset (overall) | IoU (%): 37.4 -> 51.4 | Helped |
| Use LangSplat instead of 3D-OVS | 3D semantic segmentation · 3D-OVS dataset (overall) | mIoU (%): 86.8 -> 93.4 | Helped |
| Add SAM-based hierarchical semantics (ablation) | LangSplat · ramen scene | IoU (%): 28.20 -> 46.74 | Helped |
| Add SAM-based hierarchical semantics (ablation) | LangSplat · 3D-OVS bench scene | mIoU (%): 53.2 -> 85.7 | Helped |
| Add scene-specific autoencoder and 3D Gaussian Splatting (ablation) | LangSplat with SAM · ramen scene | IoU (%): 46.74 -> 51.15 | Helped |
| Add scene-specific autoencoder and 3D Gaussian Splatting (ablation) | LangSplat with SAM · 3D-OVS bench scene | mIoU (%): 85.7 -> 94.2 | Helped |
| Remove SAM-based hierarchical semantics while retaining AE and 3D-GS (ablation) | LangSplat · ramen scene | IoU (%) | Hurt |
| Remove SAM-based hierarchical semantics while retaining AE and 3D-GS (ablation) | LangSplat · 3D-OVS bench scene | mIoU (%) | Hurt |
Surprising findings
- Naively replacing NeRF with 3D Gaussian Splatting while storing full 512-dimensional CLIP features runs out of memory, so the autoencoder is required.
Limitations
- Decoder dominates query time; the authors suggest a 1x1 convolution replacement for speed but do not implement it.
- Per-scene optimization is required, and the method is not generalized across scenes.
Experimental setup
Datasets: LERF dataset 3D-OVS dataset
Benchmarks: LERF dataset 3D-OVS dataset
Metrics: localization accuracy IoU mIoU query speed (s/query)
Baselines: LERF LSeg 3D-OVS ODISE OV-Seg FFD
Builds on: 3D Gaussian Splatting LERF Segment Anything CLIP
Closest papers by meaning
- LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPS (NeurIPS 2025 · similarity 0.93)
- Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding (CVPR 2024 · similarity 0.91)
- Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration (CVPR 2025 · similarity 0.91)
- SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting (NeurIPS 2025 · similarity 0.91)
- 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models (CVPR 2025 · similarity 0.91)
- SOVGaussian: Sparse-View 3D Gaussian Splatting for Open-Vocabulary Scene Understanding (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.