Research map / 3D vision / Human pose estimation
Unifying Correspondence Pose and NeRF for Generalized Pose-Free Novel View Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoPoNeRF, a unified pose-free generalizable NeRF framework that jointly learns 2D correspondence, relative camera pose, and NeRF rendering from a shared cost-volume representation, trained end-to-end on RealEstate10K and ACID.
Problem
Novel view synthesis from an unposed pair of images under wide baselines and minimal overlap, where pose estimation and rendering are typically treated as separate, mismatched stages.
Approach
Builds a shared 4D cosine-similarity cost volume refined by joint feature/cost aggregation and cost-volume-as-matching-distribution, feeding flow, essential-matrix-based pose, and an attention renderer jointly; uses ground-truth poses injected into rendering/projection during training (teacher-forcing-like) so pose loss gets clean gradients, plus a triplet cycle-consistency loss.
Key results
- On RealEstate10K small-overlap pose estimation, rotation average error 5.471° vs. 11.883° for FlowCAM and 17.520° for DBARF.
- Average novel view synthesis on RealEstate10K reaches PSNR 19.536 vs. 18.242 for FlowCAM and 14.789 for DBARF (pose-free setting).
- Using ground-truth pose at test time, Du et al. (pixelNeRF-style, Du et al. [20]) still outperforms the method on PSNR (21.833 vs 19.536 on RealEstate10K), while the method's learned representation with GT pose reaches 22.781 PSNR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CoPoNeRF instead of FlowCAM | pose-free novel view synthesis · RealEstate-10K Avg | PSNR: 18.242 -> 19.536 | Helped |
| use CoPoNeRF instead of FlowCAM | pose-free novel view synthesis · ACID Avg | PSNR: 20.116 -> 22.440 | Helped |
| use CoPoNeRF instead of COLMAP SP+SG | relative pose estimation · RealEstate-10K Avg | rotation error: 5.605 -> 3.610 | Helped |
| use CoPoNeRF instead of COLMAP SP+SG | relative pose estimation · ACID Avg | rotation error: 4.819 -> 3.283 | Helped |
| add pose loss (ablation) | RealEstate10K baseline · RealEstate10K Avg | PSNR: 14.646 -> 16.03 | Helped |
| add a flow head with SSIM loss (ablation) | RealEstate10K baseline · RealEstate10K Avg | PSNR: 16.03 -> 17.393 | Helped |
| add cycle loss (ablation) | RealEstate10K baseline · RealEstate10K Avg | PSNR: 17.393 -> 17.899 | Helped |
| add aggregation module (ablation) | RealEstate10K baseline · RealEstate10K Avg | PSNR: 17.899 -> 18.629 | Helped |
| add matching distribution (ablation) | RealEstate10K baseline · RealEstate10K Avg | PSNR: 18.629 -> 19.536 | Helped |
| use ground-truth poses during training (teacher forcing) (ablation) | CoPoNeRF · RealEstate-10K Avg | PSNR: 18.785 -> 19.536 | Helped |
| use ground-truth poses during training (teacher forcing) (ablation) | Baseline · RealEstate-10K Avg | PSNR: 13.856 -> 14.646 | Helped |
| use the unified CoPoNeRF representation rather than Du et al.'s representation (ablation) | novel view rendering with noisy pose · RealEstate-10K Avg | PSNR: 18.850 -> 19.500 | Helped |
Surprising findings
- Removing teacher forcing (using estimated poses during training) substantially degrades results, e.g., translation error rises from 12.766 to 40.571 degrees on RealEstate10K.
- Adding direct pose supervision to existing methods like DBARF and FlowCAM gave marginal or negative gains, unlike the proposed framework.
Limitations
- Performance lags PDC-Net+ and classical solvers on pose estimation because those use ground-truth depth-derived correspondences or precise solvers.
- Translation scale is indeterminable in two-view pose estimation, so translation evaluation is limited.
- Relies on known camera intrinsics and assumes video-derived poses for supervision.
Experimental setup
Datasets: RealEstate10K ACID
Benchmarks: RealEstate10K ACID
Metrics: PSNR SSIM LPIPS MSE rotation error (geodesic) translation angular error
Baselines: PixelNeRF Du et al. (generalizable NeRF) DBARF FlowCAM Rockwell et al. (8-point transformer) RelPose SP+SG PDC-Net+
Builds on: pixelNeRF Du et al. generalizable NeRF (epipolar attention) LoFTR-style essential matrix module (Rockwell et al.) DBARF FlowCAM PDC-Net+
Closest papers by meaning
- Free3D: Consistent Novel View Synthesis without 3D Representation (CVPR 2024 · similarity 0.88)
- NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images (CVPR 2025 · similarity 0.88)
- G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images (CVPR 2024 · similarity 0.87)
- Generalizable Novel-View Synthesis using a Stereo Camera (CVPR 2024 · similarity 0.87)
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting for Novel View Synthesis (ICML 2025 · similarity 0.87)
- From Correspondences to Pose: Non-minimal Certifiably Optimal Relative Pose without Disambiguation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.