Research map / 3D vision / NeRF and radiance fields
Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates open-vocabulary radiance-field segmentation as 3D volume segmentation, supervising CLIP embeddings directly at 3D sample points with scale-free SAM-masked embeddings, and transfers the learned language field to 3DGS for real-time rendering.
Problem
Open-vocabulary semantic understanding of NeRFs and 3D Gaussian Splatting is limited to rendered 2D masks, which do not represent the full 3D volume; prior language-embedded 3DGS methods are slow or compress features heavily.
Approach
Supervise language embeddings at 3D sample points (point-wise semantic loss) rather than rendered pixels, use SAM-masked scale-free CLIP targets, freeze the learned field's queried embeddings into 3DGS Gaussian centers initialized from the NeRF point cloud, and evaluate 3D segmentation by mesh F1-score.
Key results
- Ours-NeRF reaches 0.1520 3D F1 on Replica vs 0.0845 for LERF; Ours-3DGS reaches 0.1353.
- Ours-3DGS renders language features at 70.1 FPS (real-time), about 28x faster than LEGaussians (2.50 FPS).
- Ours-3DGS achieves 44.37 mIoU on LERF and 77.51 mIoU on 3D-OVS, versus 37.53 for LangSplat on LERF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use point-wise semantic loss instead of rendered 2D language loss (ablation) | Ours-NeRF · Replica | F1-score: 0.0537 -> 0.1532; +0.0995 | Helped |
| Use point-wise semantic loss instead of rendered 2D language loss (ablation) | Ours-NeRF · LERF | mIoU: 44.50 -> 46.37 | Helped |
| Use point-wise semantic loss instead of rendered 2D language loss (ablation) | Ours-NeRF · LERF | mAP: 44.05 -> 45.86 | Helped |
| Use scale-free embeddings instead of multi-scale embeddings (ablation) | Ours-NeRF · Replica | F1-score: 0.1114 -> 0.1532 | Helped |
| Use scale-free embeddings instead of multi-scale embeddings (ablation) | Ours-NeRF · LERF | mIoU: 30.03 -> 46.37 | Helped |
| Use scale-free embeddings instead of multi-scale embeddings (ablation) | Ours-NeRF · LERF | mAP: 28.85 -> 45.86 | Helped |
| Initialize 3DGS from the learned NeRF point cloud rather than SfM (ablation) | Ours-3DGS · Replica | F1-score: 0.1108 -> 0.1354 | Helped |
| Initialize 3DGS from the learned NeRF point cloud rather than SfM (ablation) | Ours-3DGS · LERF | mIoU: 36.14 -> 44.37 | Helped |
| Initialize 3DGS from the learned NeRF point cloud rather than SfM (ablation) | Ours-3DGS · LERF | mAP: 36.74 -> 44.57 | Helped |
| Apply post-processing to LangSplat (ablation) | LangSplat · LERF | mIoU: 32.01 -> 37.53 | Helped |
| Apply post-processing to LangSplat (ablation) | LangSplat · LERF | mAP: 30.64 -> 36.39 | Helped |
Surprising findings
- Removing the point-wise semantic loss and using the standard 2D CLIP loss cuts 3D F1 from 0.1532 to 0.0537 in the ablation.
- Ablating the LangSplat post-processing reduces its LERF mIoU from 37.53 to 32.01, showing its reported numbers depend heavily on post-processing.
Limitations
- 3DGS variant relies on a NeRF stage (40 min) before transfer, so total training is dominated by NeRF training.
- 3D-OVS evaluation restricted to four scenes with successful reconstruction.
- Ground-truth LERF masks were built by SAM with manual correction, so labels may be imperfect.
Experimental setup
Datasets: LERF 3D-OVS Replica ScanNet
Benchmarks: Replica LERF 3D-OVS ScanNet
Metrics: F1-score mIoU mAP FPS training time
Baselines: LERF OpenNeRF LEGaussians LangSplat
Builds on: LERF 3D Gaussian Splatting Nerfstudio Segment Anything
Closest papers by meaning
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.89)
- FastLGS: Speeding Up Language Embedded Gaussians with Feature Grid Mapping (AAAI 2025 · similarity 0.89)
- Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.89)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.89)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.89)
- Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.