Research map / 3D vision / 3D generation
Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Language Embedded 3D Gaussians, which quantizes CLIP+DINO language features into a discrete learned codebook and embeds compact indices in 3D Gaussians with uncertainty-guided spatial smoothing for real-time open-vocabulary 3D querying.
Problem
Open-vocabulary querying in 3D scenes requires language-embedded representations, but embedding high-dimensional language features into 3D Gaussians causes prohibitive memory use and degraded accuracy, and multi-view feature inconsistencies cause semantic ambiguity.
Approach
Vector-quantize dense CLIP+DINO features into a small discrete feature space, store compact continuous semantic vectors on 3D Gaussians decoded by a tiny MLP to index logits, and apply an uncertainty-weighted smoothing loss using a low-frequency positional-encoding MLP over Gaussian positions.
Key results
- Achieves PSNR 29.826 and mAP 0.815 on the annotated Mip-NeRF360 scenes, versus 25.749 PSNR / 0.688 mAP for LERF and 25.782 / 0.550 for 3DOVS.
- Renders at 89 FPS, compared with 0.04 FPS for LERF and 0.17 FPS for 3DOVS, and stores features in 15MB versus 320MB for LERF and 205GB for 3DOVS.
- Ablations show removing the load balancing loss drops mAP from 0.815 to 0.738, and removing DINO features drops mAP to 0.676.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Language Embedded 3D Gaussians instead of LERF | Language Embedded 3D Gaussians · Mip-NeRF360 annotated scenes | mAP: 0.688 -> 0.815 | Helped |
| use Language Embedded 3D Gaussians instead of 3DOVS | Language Embedded 3D Gaussians · Mip-NeRF360 annotated scenes | PSNR: 25.782 -> 29.826 | Helped |
| use Language Embedded 3D Gaussians instead of LERF | Language Embedded 3D Gaussians · Mip-NeRF360 annotated scenes | FPS: 0.04 -> 89 | Helped |
| add DINO features (ablation) | Language Embedded 3D Gaussians · Mip-NeRF360 annotated scenes | mAP: 0.676 -> 0.815 | Helped |
| add load balancing loss (ablation) | Language Embedded 3D Gaussians · Mip-NeRF360 annotated scenes | mAP: 0.738 -> 0.815 | Helped |
| remove learned uncertainty (ablation) | Language Embedded 3D Gaussians · Mip-NeRF360 annotated scenes | mAP: 0.773 -> 0.815 | Hurt |
| remove MLP spatial smoothing (ablation) | Language Embedded 3D Gaussians · Mip-NeRF360 annotated scenes | mAP: 0.774 -> 0.815 | Hurt |
| remove learned uncertainty and MLP smoothing (ablation) | Language Embedded 3D Gaussians · Mip-NeRF360 annotated scenes | mAP: 0.774 -> 0.815 | Hurt |
Limitations
- DINO features struggle to pinpoint fine-grained object geometry at high resolution when using CLIP-derived semantics.
- Vector quantization may lose finer-grained semantic information due to clustering of semantic concepts.
- Highly reflective or translucent objects such as televisions and mirrors are hard to detect.
Experimental setup
Datasets: Mip-NeRF360 LeRF dataset
Benchmarks: Mip-NeRF360 (six scenes with manual annotations) LeRF dataset
Metrics: PSNR SSIM LPIPS mPA mP mIoU mAP FPS memory storage
Baselines: DFF LERF 3DOVS
Builds on: 3D Gaussian Splatting VQ-VAE Switch Transformer CLIP DINO NeRF positional encoding
Closest papers by meaning
- SOVGaussian: Sparse-View 3D Gaussian Splatting for Open-Vocabulary Scene Understanding (AAAI 2025 · similarity 0.92)
- Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration (CVPR 2025 · similarity 0.92)
- LangSplat: 3D Language Gaussian Splatting (CVPR 2024 · similarity 0.91)
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.91)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.90)
- PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understanding (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.