Research map / 3D vision / NeRF and radiance fields
GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GSNeRF, a generalizable semantic NeRF that predicts source/target depth to guide sampling, jointly rendering novel-view RGB and semantic maps for unseen scenes.
Problem
Generalizable novel view synthesis combined with semantic segmentation for unseen scenes, where existing semantic NeRFs either require per-scene retraining or sample inefficiently and aggregate noisy features along rays.
Approach
A Semantic Geo-Reasoner predicts depth for each source view and aggregates them into a target-view depth map; this depth guides Gaussian-distributed sampling for volume rendering and a single surface-point query for semantic rendering, avoiding noisy off-surface points.
Key results
- On ScanNet with GT depth-free training, GSNeRF achieves 52.21 mIoU and 31.49 PSNR versus 47.69 mIoU and 25.13 PSNR for S-Ray.
- On Replica, GSNeRF reaches 51.23 mIoU and 31.71 PSNR, outperforming S-Ray (43.27 mIoU, 26.77 PSNR) in the no-GT-depth setting.
- With only 4 sampled points per ray, GSNeRF keeps PSNR near 28 on ScanNet, while competitors degrade substantially.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use depth-guided sampling and semantic rendering rather than uniform sampling with a pretrained 2D segmentor (ablation) | GSNeRF · ScanNet | PSNR: 28.48 -> 31.49 | Helped |
| Use depth-guided sampling and semantic rendering rather than uniform sampling with a pretrained 2D segmentor (ablation) | GSNeRF · ScanNet | mIoU: 37.88 -> 52.21 | Helped |
| Use depth-guided sampling for image rendering instead of uniform sampling (ablation) | GSNeRF · ScanNet | PSNR: 28.48 -> 30.56 | Helped |
| Add the semantic renderer instead of relying on a pretrained 2D segmentor (ablation) | GSNeRF · ScanNet | mIoU: 41.72 -> 49.47 | Helped |
| Use depth-guided sampling for semantic surface-point rendering instead of uniform sampling (ablation) | GSNeRF · ScanNet | mIoU: 49.47 -> 52.21 | Helped |
| Use depth-guided sampling for semantic surface-point rendering instead of uniform sampling (ablation) | GSNeRF · ScanNet | PSNR: 30.88 -> 31.49 | Helped |
| Use GSNeRF rather than S-Ray | generalizable semantic NeRFs · ScanNet, with GT depth | mIoU: 55.53 -> 58.30 | Helped |
| Use GSNeRF rather than GeoNeRF + semhead | generalizable semantic NeRFs · ScanNet, with GT depth | PSNR: 32.55 -> 31.33 | Hurt |
| Train GSNeRF without GT depth, using self-supervised depth loss | GSNeRF · ScanNet | mIoU: 47.69 -> 52.21 | Helped |
| Train GSNeRF without GT depth, using self-supervised depth loss | GSNeRF · Replica | mIoU: 43.27 -> 51.23 | Helped |
Surprising findings
- GeoNeRF + semhead with GT depth supervision slightly outperforms GSNeRF on PSNR and SSIM for RGB on ScanNet, though GSNeRF leads on semantic metrics.
Limitations
- Semantic rendering uses only one surface point per ray, relying on the accuracy of predicted target depth.
- Compared with GeoNeRF with GT depth supervision, PSNR and SSIM on ScanNet are slightly lower.
Experimental setup
Datasets: ScanNet Replica
Benchmarks: ScanNet Replica
Metrics: mIoU accuracy class accuracy PSNR SSIM LPIPS
Baselines: S-Ray GeoNeRF + semhead Neuray + semhead MVSNeRF + semhead GNT + semhead Semantic NeRF
Builds on: GeoNeRF Neuray S-Ray DS-NeRF RC-MVSNet
Closest papers by meaning
- GSN: Generalisable Segmentation in Neural Radiance Field (AAAI 2024 · similarity 0.91)
- GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding (CVPR 2024 · similarity 0.89)
- Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation (ICLR 2024 · similarity 0.89)
- ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field (AAAI 2024 · similarity 0.89)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.88)
- Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.