Research map / 3D vision / NeRF and radiance fields
GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GP-NeRF, which co-aggregates radiance and semantic-embedding fields with transformers, renders semantic features (not labels) jointly, and feeds them to a U-Net perception head, trained with two self-distillation losses for context-aware 3D segmentation.
Problem
Per-pixel semantic/instance label rendering in NeRFs ignores image context, causing unclear boundaries and inconsistent segmentation within objects; existing generalized semantic NeRFs also fail to jointly optimize RGB and semantic geometry.
Approach
Treat segmentation as context-aware prediction: transformer-based Field-Aggregation and Ray-Aggregation modules jointly build radiance and semantic-embedding fields, render semantic features into a novel-view 2D map fed to a segmentation head, and use a gradient-blocked 2D semantic distillation loss plus a depth-guided per-point semantic loss to prevent cheating.
Key results
- On Replica generalized semantic segmentation, GP-NeRF reaches 48.53 mIoU vs 41.59 for Semantic-Ray (reported as +6.94%).
- Instance segmentation AP75 on Replica averages 89.50 vs 81.03 for DM-NeRF (+8.47%).
- On ScanNet generalized setting, GP-NeRF attains 59.92 mIoU and 29.37 PSNR, surpassing Semantic-Ray's 57.15 mIoU and 26.57 PSNR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GP-NeRF instead of Semantic-Ray | generalized semantic segmentation · Replica | mIoU: +6.94% | Helped |
| use GP-NeRF instead of Semantic-Ray | generalized semantic segmentation · ScanNet | mIoU: +2.7% | Helped |
| use GP-NeRFft instead of S-Rayft | finetuned semantic segmentation · Replica | mIoU: +11.76% | Helped |
| use GP-NeRFft instead of Semantic-NeRF | finetuned semantic segmentation · ScanNet | mIoU: +2.6% | Helped |
| use GP-NeRF instead of DM-NeRF | instance segmentation · Replica | AP0.75: +8.47% | Helped |
| use GP-NeRF instead of Semantic-Ray | generalized reconstruction · ScanNet | PSNR: 2.8% | Helped |
| jointly optimize attention maps instead of freezing them (ablation) | GP-NeRF · ScanNet | mIoU: 56.45 -> 57.19 | Helped |
| jointly optimize attention maps instead of freezing them (ablation) | GP-NeRF · ScanNet | PSNR: 29.06 -> 29.30 | Helped |
| add 2D Semantic Distill Loss without Gradient Block (ablation) | GP-NeRF · ScanNet | mIoU: 57.19 -> 52.03 | Hurt |
| add Gradient Block with 2D Semantic Distill Loss (ablation) | GP-NeRF · ScanNet | mIoU: 52.03 -> 59.55 | Helped |
| add Depth-Guided Semantic Distill Loss (ablation) | GP-NeRF with 2D Semantic Distill Loss and Gradient Block · ScanNet | mIoU: +0.37% | Helped |
| add Depth-Guided Semantic Distill Loss (ablation) | GP-NeRF with 2D Semantic Distill Loss and Gradient Block · ScanNet | PSNR: +0.11% | Helped |
Surprising findings
- Applying the 2D semantic distillation loss without a gradient block drops mIoU by about 5 points, because shared encoder layers learn less discriminative features to 'cheat' the loss.
- The 2D semantic distillation loss slightly hurts reconstruction PSNR (about 0.4 dB) unless paired with the depth-guided loss.
Limitations
- Improvements on ScanNet are smaller than on Replica (2.7% vs 6.94% generalized mIoU).
- The 2D distillation loss can cause the rendered features to cheat unless gradient blocking and depth guidance are used.
Experimental setup
Datasets: Replica ScanNet
Benchmarks: Replica ScanNet
Metrics: mIoU total accuracy average accuracy AP PSNR SSIM LPIPS
Baselines: Semantic-Ray NeuRay + Semantic Head MVSNeRF + Semantic Head Semantic-NeRF DM-NeRF GNT Mask R-CNN Swin-T
Builds on: GNT Semantic-Ray NeuRay Semantic-NeRF DM-NeRF
Closest papers by meaning
- Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (CVPR 2024 · similarity 0.90)
- GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding (CVPR 2024 · similarity 0.89)
- NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs (CVPR 2024 · similarity 0.88)
- Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space (AAAI 2025 · similarity 0.88)
- Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling (CVPR 2024 · similarity 0.87)
- SANeRF-HQ: Segment Anything for NeRF in High Quality (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.