Research map / 3D vision / Gaussian splatting
Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Dr. Splat registers CLIP embeddings directly onto 3D Gaussians (no rendering) and compresses them with a pre-trained Product Quantization codebook for fast open-vocabulary 3D search.
Problem
Language-embedded 3D Gaussian Splatting methods distill CLIP features via rendered 2D feature maps, causing embedding distortion, slow multi-view search, and per-scene compression that hinders generalization for 3D localization and segmentation.
Approach
Directly assign SAM-mask CLIP embeddings to the top-k Gaussians along each pixel ray via weight-averaged aggregation (an inverse volume rendering without gradient optimization), and store each Gaussian's feature as a PQ index from a codebook trained once on LVIS CLIP patch embeddings.
Key results
- On LeRF-OVS 3D object selection, Dr. Splat (Top-40) reaches 39.07 mIoU vs. 34.60 for OpenGaussian and 8.29 for LangSplat-m.
- On ScanNet 3D object localization (19 classes), Dr. Splat (Top-40) achieves 25.4 mIoU and 25.6 IoU>0.45 vs. 25.2 and 18.3 for OpenGaussian.
- On ScanNet 3D semantic segmentation (10 classes), Dr. Splat (Top-40) achieves 50.2 mIoU vs. 49.7 for OpenGaussian, with lower gains on 19 classes (29.6 vs. 30.1 for OpenGaussian).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use direct 3D feature registration with Top-40 Gaussians | Dr. Splat · LERF-OVS | mIoU: 43.06 -> 43.58 | Helped |
| Use direct 3D feature registration with Top-20 Gaussians | Dr. Splat · LERF-OVS | mAcc @ 0.25: 59.61 -> 64.32 | Helped |
| Use direct 3D feature registration with Top-40 Gaussians | Dr. Splat · ScanNet | 3D localization IoU > 0.45: 18.3 -> 25.6 | Helped |
| Use direct 3D feature registration with Top-40 Gaussians | Dr. Splat · ScanNet, 19 classes | mIoU: 30.1 -> 29.6 | Hurt |
| Use direct 3D feature registration with Top-40 Gaussians | Dr. Splat · ScanNet, 10 classes | mIoU: 49.7 -> 50.2 | Helped |
| Increase the number of Gaussians assigned per ray (ablation) | Dr. Splat · ScanNet | 3D localization performance | Helped |
| Increase the number of Gaussians assigned per ray (ablation) | Dr. Splat · ScanNet | memory consumption | Hurt |
| Vary PQ sub-vector size among 64, 128, and 256 (ablation) | Dr. Splat · ScanNet | localization quality | Mixed |
Surprising findings
- On 19-class ScanNet segmentation, Dr. Splat (Top-20) and (Top-40) are at or below OpenGaussian in mIoU (28.0 and 29.6 vs. 30.1).
Limitations
- Relies on LangSplat-style hyperparameters and only one OpenGaussian comparison as a fair baseline; rasterization baselines were modified by the authors.
- Performance gains over OpenGaussian are small on some ScanNet settings (e.g., 19-class segmentation mIoU and 10-class mIoU).
- Evaluated on only eight randomly selected ScanNet scenes and the LeRF-OVS dataset.
Experimental setup
Datasets: LVIS ScanNet LeRF-OVS
Benchmarks: LeRF-OVS ScanNet
Metrics: mIoU mAcc@0.25 volume-aware IoU at thresholds 0.15/0.3/0.45 mAcc
Baselines: LangSplat-m LEGaussians-m OpenGaussian
Builds on: 3D Gaussian Splatting LangSplat LeRF Product Quantization
Closest papers by meaning
- Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding (CVPR 2024 · similarity 0.92)
- Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting (NeurIPS 2025 · similarity 0.92)
- SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting (NeurIPS 2025 · similarity 0.91)
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.91)
- 3D Vision-Language Gaussian Splatting (ICLR 2025 · similarity 0.91)
- LangSplat: 3D Language Gaussian Splatting (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.