Research map / 3D vision / NeRF and radiance fields
PNeRFLoc: Visual Localization with Point-Based Neural Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PNeRFLoc, a visual localization framework that combines structure-based 2D-3D matching on a point-based NeRF (PointNeRF) with efficient rendering-based pose refinement via a warping loss.
Problem
Visual localization in known scenes: existing NeRF-boosted localization methods use NeRF only for data augmentation for regression, limiting accuracy on novel viewpoints and appearances due to lack of geometric constraints.
Approach
A unified point-based radiance field whose scene-specific feature adaptation MLP bridges scene-agnostic R2D2 localization features and NeRF rendering features, enabling both 2D-3D PnP initialization and rendering-based pose refinement that minimizes a warping loss (rendering only once, no backprop through the network).
Key results
- On Replica, PNeRFLoc achieves median errors such as 0.005 m / 0.29° on room0, outperforming PixLoc (0.055 m / 1.89°) and CoordiNet.
- On 7Scenes, PNeRFLoc matches or is slightly worse than PixLoc (e.g., 0.02 m / 0.80° on Chess for both), while outperforming other methods generally.
- On Cambridge Landmarks, PNeRFLoc is on par with SOTA and outperforms other NeRF-boosted methods such as LENS and DFNet (e.g., Kings 0.24 m / 0.29° vs PixLoc 0.14 m / 0.24°).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PNeRFLoc instead of PixLoc | visual localization · Replica room0 | median translation/rotation error: 0.055/1.89 -> 0.005/0.29 | Helped |
| use PNeRFLoc instead of PixLoc | visual localization · 7Scenes Chess | median translation/rotation error: 0.02/0.80 -> 0.02/0.80 | No clear effect |
| use PNeRFLoc instead of PixLoc | visual localization · Cambridge Kings | median translation/rotation error: 0.14/0.24 -> 0.24/0.29 | Hurt |
| add rendering-based optimization to structure-based initialization (ablation) | PNeRFLoc · Replica room0 and office0 | median translation/rotation error: 0.030 / 0.79 -> 0.005 / 0.29; 0.082 / 1.38 -> 0.006 / 0.56 | Helped |
| add a blank depth mask (ablation) | PNeRFLoc rendering-based optimization · Replica room0 and office0 | median translation/rotation error: 0.027 / 0.63 -> 0.005 / 0.29; 0.050 / 1.09 -> 0.006 / 0.56 | Helped |
| use warping loss instead of photometric loss (ablation) | PNeRFLoc rendering-based optimization · Replica room0 and office0 | median translation/rotation error: 0.035 / 0.81 -> 0.005 / 0.29; 0.082 / 1.38 -> 0.006 / 0.56 | Helped |
| use GT depth instead of estimated depth (ablation) | PNeRFLoc · 7Scenes chess | median translation/rotation error: 0.02/0.80 -> 0.02/0.80 | No clear effect |
| use GT depth instead of estimated depth (ablation) | PNeRFLoc · 7Scenes office | median translation/rotation error: 0.03/1.05 -> 0.03/1.06 | Mixed |
| use GT depth instead of estimated depth (ablation) | PNeRFLoc · 7Scenes stairs | median translation/rotation error: 0.32/5.73 -> 0.20/3.61 | Helped |
| use GT depth instead of estimated depth (ablation) | PNeRFLoc · Replica room1, office0, and office1 | median translation/rotation error: 0.02/0.55 -> 0.01/0.34; 0.01/0.56 -> 0.01/0.54; 0.02/0.64 -> 0.01/0.39 | Helped |
Surprising findings
- Rendering-based optimization with the warping loss takes about 5.5 s per image versus about 47 s with photometric loss, while being more accurate.
- Using estimated depth instead of ground-truth depth yields nearly identical localization accuracy on real-world 7Scenes, since noisy Kinect depth limits NeRF quality.
Limitations
- Performance depends on accurate depth and camera poses; inaccurate poses on Cambridge limit NeRF quality.
- Not yet real-time; efficiency still limits use in visual odometry applications.
- Performs on par or slightly worse than PixLoc on real-world 7Scenes and Cambridge datasets.
Experimental setup
Datasets: Cambridge Landmarks 7Scenes Replica
Benchmarks: Replica Cambridge Landmarks 7Scenes
Metrics: median translation error (m) median rotation error (degrees) time per image (s) recall at (5cm, 5 degrees)
Baselines: PixLoc CoordiNet PoseNet LENS DFNet DSAC*
Builds on: PointNeRF R2D2 NeRF-W iNeRF PixLoc
Closest papers by meaning
- ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field (AAAI 2024 · similarity 0.85)
- CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning (AAAI 2024 · similarity 0.85)
- ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Field (NeurIPS 2024 · similarity 0.85)
- Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation (ICLR 2024 · similarity 0.84)
- R-SCoRe: Revisiting Scene Coordinate Regression for Robust Large-Scale Visual Localization (CVPR 2025 · similarity 0.84)
- Enhancing Visual Localization with Cross-Domain Image Generation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.