Research map / Point clouds / Point cloud classification
OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OpenNeRF, a NeRF that distills pixel-aligned OpenSeg features into an open-set field and uses uncertainty-guided novel view rendering to improve open-vocabulary 3D semantic segmentation on Replica.
Problem
Open-set 3D scene segmentation is limited by mesh/point-cloud resolution and by global CLIP features that are poorly localized in 3D, as in LERF and OpenScene.
Approach
Encode pixel-wise OpenSeg VLM features directly into a Mip-NeRF-based radiance field (no multi-scale patches or DINO regularization), and select novel camera views at points with high multi-view feature uncertainty (generalized variance) to extract additional open-set features.
Key results
- On Replica 3D segmentation, OpenNeRF reaches 20.4 mIoU vs 15.9 for OpenScene (Ensemble) and 10.5 for LERF.
- Head/common/tail mIoU of 35.4/20.1/5.8 for OpenNeRF, versus 31.7/14.5/1.5 for OpenScene Ensemble.
- Replacing novel views with random cameras drops performance to 15.4 mIoU, while uncertainty-guided novel views reach 20.4 mIoU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OpenNeRF instead of OpenScene Ensemble | Replica 3D semantic segmentation · Replica All | mIoU: +4.5 mIoU | Helped |
| Use OpenNeRF instead of OpenScene Ensemble | Replica 3D semantic segmentation · Replica Head | mIoU: +3.7 | Helped |
| Use OpenNeRF instead of OpenScene Ensemble | Replica 3D semantic segmentation · Replica Common | mIoU: +5.6 | Helped |
| Use OpenNeRF instead of OpenScene Ensemble | Replica 3D semantic segmentation · Replica Tail | mIoU: +4.3 | Helped |
| Replace OpenScene Distilled with OpenScene Ensemble | OpenScene · Replica All | mIoU: +1.1 mIoU | Helped |
| Render and project features rather than sampling a single 3D point (ablation) | OpenNeRF · Replica | mIoU: 16.5 -> 18.5 | Helped |
| Add depth supervision (ablation) | OpenNeRF · Replica | mIoU: 18.5 -> 19.4 | Helped |
| Add rendered novel views selected by uncertainty (ablation) | OpenNeRF · Replica | mIoU: 19.4 -> 20.4 | Helped |
| Add views along the original camera trajectory instead of selected novel views (ablation) | OpenNeRF · Replica | mIoU | No clear effect |
| Add random camera positions in the scene instead of selected novel views (ablation) | OpenNeRF · Replica | mIoU: 15.4 mIoU | Hurt |
Surprising findings
- Random extra camera views degrade performance substantially (15.4 mIoU), below the non-novel-view variant, showing that poorly placed views harm open-set features.
- Sampling features directly from the NeRF already outperforms OpenScene and LERF before any novel views are added.
Limitations
- Many long-tail classes are not detected by either method on Replica.
- Depth supervision is optional and depends on available depth data.
- Novel view selection relies on a coarse point cloud reconstruction and uncertainty heuristics.
Experimental setup
Datasets: Replica Nice-SLAM RGB-D sequences
Benchmarks: Replica
Metrics: mIoU mAcc
Baselines: LERF OpenScene (Distilled) OpenScene (Ensemble)
Builds on: Mip-NeRF OpenSeg LERF OpenScene
Closest papers by meaning
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.89)
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.89)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.88)
- Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model (CVPR 2025 · similarity 0.88)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.87)
- GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.