Research map / 3D vision / 3D generation
GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GOV-NeSF, a generalizable neural semantic field that blends multi-view colors and LSeg open-vocabulary features via a cross-view-attention FusionNet, trained on 2D posed images (ScanNet) without 3D labels or depth.
Problem
Open-vocabulary 3D scene understanding methods lack generalization to unseen scenes, require 3D data or per-scene optimization, and naive multi-view feature averaging or direct 2D-to-3D distillation degrades performance.
Approach
Instead of regressing open-vocabulary features directly, the model learns view-specific blending weights over multi-view OV features (from LSeg) and colors, predicted by a FusionNet with cross-view attention over a geometry-aware 3D cost volume, and rendered via volume rendering supervised by novel views.
Key results
- On Replica 2D open-vocabulary segmentation from novel views, GOV-NeSF reaches 44.3 mIoU versus 9.7 for S-Ray-OV and 5.0 for the Distill Baseline when trained only on ScanNet.
- On ScanNet 3D open-vocabulary segmentation without depth, GOV-NeSF gets 45.7 mIoU vs 42.1 for OpenScene-2D w/o Depth; on Replica 32.8 vs 31.2.
- With depth-guided masking, 3D mIoU on ScanNet rises to 53.5, slightly above OpenScene-2D with depth (52.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Multi-view Joint Fusion instead of direct open-vocabulary feature regression (ablation) | GOV-NeSF · ScanNet | 2D mIoU: 46.4 -> 52.2 | Helped |
| Use Multi-view Joint Fusion instead of direct open-vocabulary feature regression | GOV-NeSF · Replica | 2D mIoU: 5.0 -> 44.3 | Helped |
| Add Volume Feature Aggregation (ablation) | GOV-NeSF · ScanNet | 2D mIoU: 49.5 -> 52.2 | Helped |
| Add Cross-View Attention (ablation) | GOV-NeSF · ScanNet | 2D mIoU: 50.7 -> 52.2 | Helped |
| Add Depth-Guided Masking (ablation) | GOV-NeSF · ScanNet | 2D mIoU: 52.2 -> 46.8 | Hurt |
| Add Depth-Guided Masking (ablation) | GOV-NeSF · ScanNet | 3D mIoU: 45.7 -> 53.5 | Helped |
| Add Depth-Guided Masking | GOV-NeSF · Replica | 3D mIoU: 35.8 -> 37.7 | Helped |
| Use learned multi-view blending instead of naive multi-view feature averaging | GOV-NeSF · Replica, 3D w/o depth | 3D mIoU: 31.2 -> 32.8 | Helped |
| Use the generalizable GOV-NeSF representation rather than OpenScene-3D | GOV-NeSF · Replica, 3D segmentation | 3D mIoU: 3.3 -> 32.8 | Helped |
| Use GOV-NeSF for novel-view rendering | GOV-NeSF · Replica novel view synthesis | PSNR: 18.1 -> 21.5 | Helped |
Surprising findings
- Direct distillation of open-vocabulary features (Distill Baseline) collapses on Replica (5.0 mIoU), showing poor cross-domain generalization.
- The depth-guided masking module improves 3D segmentation but lowers 2D segmentation mIoU (46.8 vs 52.2) due to holes in rendered images.
Limitations
- Rendered color images are blurry compared with S-Ray since no depth priors are used for room-scale scenes.
- Novel view synthesis quality lags NeuRay and S-Ray (PSNR 21.5 vs 22.0-22.8 on ScanNet) and LPIPS is worse.
- Depth-guided masking hurts 2D segmentation due to empty holes in rendered images.
Experimental setup
Datasets: ScanNet Replica
Benchmarks: ScanNet Replica
Metrics: mIoU oAcc mAcc PSNR SSIM LPIPS
Baselines: S-Ray-OV Distill Baseline OpenScene-2D OpenScene-3D LSeg NeRF-Det NeuRay S-Ray MVSNeRF
Builds on: Semantic-NeRF IBRNet S-Ray OpenScene LSeg
Closest papers by meaning
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.90)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.89)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.89)
- econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussians (ICLR 2025 · similarity 0.89)
- Open-Vocabulary 3D Semantic Segmentation with Foundation Models (CVPR 2024 · similarity 0.89)
- Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.