Research map / 3D vision / Human pose estimation
Neural Refinement for Absolute Pose Regression with Feature Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a test-time refinement pipeline for absolute pose regression that backpropagates a feature-metric error from a neural feature synthesizer (NeFeS) rendering dense features at the coarse pose, improving APR accuracy on Cambridge Landmarks and 7-Scenes.
Problem
Absolute pose regression (APR) networks predict camera pose with 2D operations only at inference, lacking 3D geometric constraints, so their accuracy is limited and they generalize poorly to viewpoints not covered by training data.
Approach
At test time, a pretrained APR's coarse pose is refined by minimizing a spatial-normalized feature cosine loss between features rendered by a NeRF-style feature field (NeFeS, with exposure-adaptive affine colour transform and a feature fusion module, trained progressively) and features from a 2D feature extractor, backpropagating into the APR weights or the pose.
Key results
- DFNet + NeFeS50 achieves average 0.35m/0.77° on Cambridge Landmarks versus DFNet's 1.19m/2.90°.
- DFNet + NeFeS50 achieves average 0.02m/0.79° on 7-Scenes versus DFNet's 0.06m/1.93°, claimed SOTA.
- Refinement reduces error for PoseNet, MS-Transformer and DFNet across datasets (e.g., Cambridge PoseNet 2.04m/6.23° to 0.54m/1.05°).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add NeFeS test-time refinement to DFNet | DFNet · Cambridge Landmarks (average) | median position/orientation error (m/°): 1.19/2.90 -> 0.35/0.77 | Helped |
| Add NeFeS test-time refinement to DFNet | DFNet · 7-Scenes (average) | median translational/rotational error (m/°): 0.06/1.93 -> 0.02/0.79 | Helped |
| Add NeFeS refinement | PoseNet · 7-Scenes | median translational/rotational error (m/°): 0.21m/7.74 -> 0.08m/2.83° | Helped |
| Add NeFeS refinement | MS-Transformer · 7-Scenes | median translational/rotational error (m/°): 0.18m/9.51° -> 0.11m/3.46° | Helped |
| Remove exposure-adaptive ACT (ablation) | NeFeS · Cambridge Shop Facade | median orientation error: 0.53° -> 1.20° | Hurt |
| Add Feature Fusion (ablation) | NeFeS without ACT · Cambridge Shop Facade | median position/orientation error (m/°): 0.37m/1.62° -> 0.15m/1.20° | Helped |
| Use progressive rather than combined training (ablation) | NeFeS · Cambridge Shop Facade | median position/orientation error (m/°): 0.17m/0.80° -> 0.15m/0.53° | Helped |
| Use spatial-wise rather than channel-wise normalization (ablation) | Feature cosine similarity loss during refinement · Cambridge Shop | median position/orientation error (m/°): 0.20m/1.05° -> 0.15m/0.53° | Helped |
| Optimize APR parameters instead of directly optimizing pose | DFNet + NeFeS50 · Cambridge | median position/orientation error (m/°): 0.66m/1.39° -> 0.35m/0.77° | Helped |
Surprising findings
- Optimizing the APR network's parameters at test time yields lower error than directly optimizing the camera pose with the same number of iterations.
- Refinement can only correct perturbations up to about 35° rotation and roughly 0.6m (indoor) or 4m (outdoor) translation.
Limitations
- Refinement fails when there is insufficient overlap between rendered and query features; error bounds around 35° rotation.
- Requires per-scene NeFeS training and per-image iterative optimization at test time, adding inference cost versus a single forward pass.
- Relies on SfM ground-truth poses; dSLAM GT yields poor NeRF renderings on 7-Scenes.
Experimental setup
Datasets: Cambridge Landmarks 7-Scenes
Benchmarks: Cambridge Landmarks 7-Scenes
Metrics: median translation error (m) median rotation error (°)
Baselines: PoseNet MapNet MS-Transformer PAE DFNet TransPoseNet LSTM PN
Builds on: NeRF-W DFNet Direct-PN URF feature field distillation
Closest papers by meaning
- Map-Relative Pose Regression for Visual Re-Localization (CVPR 2024 · similarity 0.87)
- ConDo: Continual Domain Expansion for Absolute Pose Regression (AAAI 2025 · similarity 0.86)
- ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization (CVPR 2024 · similarity 0.85)
- PORF: POSE RESIDUAL FIELD FOR ACCURATE NEURAL SURFACE RECONSTRUCTION (ICLR 2024 · similarity 0.85)
- Scene-agnostic Pose Regression for Visual Localization (CVPR 2025 · similarity 0.85)
- The Unreasonable Effectiveness of Pre-Trained Features for Camera Pose Refinement (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.