Research map / 3D vision / Human pose estimation
PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PF-LRM, a single-stream transformer that jointly reconstructs a triplane NeRF and estimates relative camera poses from a few unposed images, using per-patch point prediction and a differentiable PnP solver.
Problem
Reconstructing 3D objects from sparse, unposed, wide-baseline images where SfM fails, requiring joint estimation of camera poses and 3D shape.
Approach
A single-stream transformer applies self-attention jointly over triplane NeRF tokens and image patch tokens; per-patch 3D points (distilled online from the NeRF geometry) form 3D-2D correspondences solved by a differentiable EPro-PnP solver, instead of regressing poses directly.
Key results
- On OmniObject3D, GSO and ABO, PF-LRM reduces average rotation error roughly 14.6x vs FORGE and 14.7x vs RelPose++ (w/o bg) with 4 sparse input views.
- On unseen OmniObject3D, GSO and ABO novel-view synthesis, PF-LRM achieves average PSNR of about 24.8 vs 13.4 for FORGE.
- Single-feed-forward inference takes ~1.3 seconds on one A100 GPU; the 590M-parameter model was trained on 128 A100s for about one week.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PF-LRM instead of the strongest listed baseline | pose estimation · OmniObject3D | R. error: 69.22 -> 6.32 | Helped |
| use PF-LRM instead of the strongest listed baseline | pose estimation · GSO | R. error: 103.81 -> 3.99 | Helped |
| use PF-LRM instead of the strongest listed baseline | pose estimation · ABO | R. error: 102.30 -> 16.27 | Helped |
| use PF-LRM instead of the strongest listed baseline | pose estimation · CO3D | R. error: 28.24 -> 15.53 | Helped |
| use PF-LRM instead of the strongest listed baseline | pose estimation · DTU | R. error: 11.84 -> 10.42 | Helped |
| use PF-LRM instead of FORGE | 3D reconstruction · OmniObject3D | PSNR: 17.95 -> 23.02 | Helped |
| use PF-LRM instead of FORGE | 3D reconstruction · GSO | PSNR: 11.43 -> 25.04 | Helped |
| use PF-LRM instead of FORGE | 3D reconstruction · ABO | PSNR: 10.92 -> 26.23 | Helped |
| use the smaller model instead of Ours (L) (ablation) | PF-LRM · GSO | R. error: 2.46 -> 13.08 | Hurt |
| remove NeRF prediction (ablation) | Ours (S) · GSO | R. error: 13.08 -> 111.89 | Hurt |
| add pose prediction (ablation) | Ours (S) · GSO | PSNR_g.t.: 22.48 -> 23.80 | Helped |
| replace differentiable PnP with CLS-token MLP pose prediction (ablation) | Ours (S) · GSO | R. error: 13.08 -> 25.32 | Hurt |
Surprising findings
- Predicting poses with a plain MLP regressor from image tokens yields nearly random poses on sparse views when trained on Objaverse, while the NeRF-supervised differentiable PnP approach works.
- Removing the NeRF prediction branch makes differentiable PnP training unstable and divergent.
Limitations
- Ignores background information that could provide camera pose cues; requires foreground masks.
- Assumes known camera intrinsics and cannot model view-dependent effects; triplane resolution limits detail.
- Requires ground-truth pose supervision during training.
Experimental setup
Datasets: Objaverse MVImgNet GSO OmniObject3D ABO CO3D DTU
Benchmarks: OmniObject3D GSO ABO CO3D DTU
Metrics: rotation error Acc@15 Acc@30 translation error PSNR SSIM LPIPS
Baselines: FORGE RelPose++ HLoc SRT
Builds on: LRM EPro-PnP DINO NeRF FORGE
Closest papers by meaning
- LRM: Large Reconstruction Model for Single Image to 3D (ICLR 2024 · similarity 0.89)
- LRM-Zero: Training Large Reconstruction Models with Synthesized Data (NeurIPS 2024 · similarity 0.87)
- LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields (CVPR 2025 · similarity 0.87)
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting for Novel View Synthesis (ICML 2025 · similarity 0.86)
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images (ICLR 2025 · similarity 0.86)
- Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.