Research map / Video / Video segmentation and tracking
PoseCrafter: Extreme Pose Estimation with Hybrid Video Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoseCrafter, a training-free pipeline that couples DynamiCrafter video interpolation with pose-conditioned ViewCrafter refinement and selects intermediate frames by ORB feature-matching inlier counts, improving DUSt3R pose estimation for small- or no-overlap image pairs.
Problem
Relative camera pose estimation from image pairs with small or no overlap, where existing video-interpolation-based methods (InterPose) produce blurry intermediate frames and use slow, statistical frame selection.
Approach
Hybrid Video Generation uses DynamiCrafter to produce coarse relay frames (start, first, second-to-last, end), estimates their poses with DUSt3R, and conditions ViewCrafter on an interpolated camera trajectory to render sharper intermediate frames; Feature Matching Selector deterministically ranks generated frames by RANSAC inlier counts against the input pair instead of self-consistency scoring.
Key results
- On Cambridge Landmarks yaw 50-65°, PoseCrafter reduces MRE from 13.61° (InterPose) to 11.40° and raises AUC30 from 70.47 to 77.41.
- On ScanNet yaw 65-90°, MRE drops from 28.22° (InterPose) to 17.02° and AUC30 rises to 56.44.
- On NAVI, MRE improves from 6.61 (InterPose) to 5.97 with AUC30 83.98 versus 82.80.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PoseCrafter instead of InterPose | DUSt3R pose estimation · Cambridge Landmarks [50°–65°] | MRE: 13.61 -> 11.40 | Helped |
| Use PoseCrafter instead of InterPose | DUSt3R pose estimation · Cambridge Landmarks [65°–90°] | MRE: 38.87 -> 29.02 | Helped |
| Use PoseCrafter instead of InterPose | DUSt3R pose estimation · ScanNet [50°–65°] | MRE: 15.15 -> 10.77 | Helped |
| Use PoseCrafter instead of InterPose | DUSt3R pose estimation · ScanNet [65°–90°] | MRE: 28.22 -> 17.02 | Helped |
| Use PoseCrafter instead of InterPose | DUSt3R pose estimation · DL3DV-10K [50°–90°] | MRE: 13.17 -> 12.73 | Helped |
| Use PoseCrafter instead of InterPose | DUSt3R pose estimation · NAVI [50°–90°] | MRE: 6.61 -> 5.97 | Helped |
| Add FMS to hybrid video generation (ablation) | PoseCrafter · Cambridge Landmarks [50°–65°] | MRE: 13.24 -> 11.40 | Helped |
| Replace FMS with SCS (ablation) | PoseCrafter · Cambridge Landmarks [50°–65°] | MRE: 11.40 -> 12.11 | Hurt |
| Combine DynamiCrafter with ViewCrafter for hybrid video generation (ablation) | PoseCrafter without FMS · Cambridge Landmarks [50°–65°] | MRE: 16.11 -> 13.24 | Helped |
| Replace DynamiCrafter with ToonCrafter (ablation) | PoseCrafter · Cambridge Landmarks [50°–65°] | MRE: 11.40 -> 12.08 | Hurt |
| Use ORB rather than RoMa for FMS (ablation) | PoseCrafter · Cambridge Landmarks [50°–65°] | MRE: 13.89 -> 11.40 | Helped |
| Use ORB rather than LoFTR for FMS (ablation) | PoseCrafter · Cambridge Landmarks [50°–65°] | MRE: 12.41 -> 11.40 | Helped |
Surprising findings
- Using ORB keypoints in FMS outperformed learned matchers RoMa, LoFTR, and SuperPoint, which the authors attribute to robustness against color blurring in synthesized frames.
- Keeping more relay frames (6, 8, 16) increased rotation error relative to keeping only 4, because central frames become blurrier.
Limitations
- Severe illumination differences between start and end frames introduce artifacts in synthesized frames that receive low confidence.
- FMS fails in scenes with uniform or repetitive textures, yielding few reliable RANSAC inliers.
- Results on the MASt3R backbone and VGGT comparisons show lower gains than with DUSt3R.
Experimental setup
Datasets: Cambridge Landmarks ScanNet DL3DV-10K NAVI
Benchmarks: Cambridge Landmarks ScanNet DL3DV-10K NAVI
Metrics: MRE MTE R@5 R@15 R@30 T@5 T@15 T@30 AUC30
Baselines: DUSt3R InterPose VGGT MASt3R
Builds on: InterPose DynamiCrafter ViewCrafter DUSt3R
Closest papers by meaning
- ExPose: Reinforcing Video Generation Models for Extreme Pose Estimation (CVPR 2026 · similarity 0.90)
- Can Generative Video Models Help Pose Estimation? (CVPR 2025 · similarity 0.88)
- Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis (NeurIPS 2024 · similarity 0.88)
- From Correspondences to Pose: Non-minimal Certifiably Optimal Relative Pose without Disambiguation (CVPR 2024 · similarity 0.85)
- NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images (CVPR 2025 · similarity 0.85)
- Unifying Correspondence Pose and NeRF for Generalized Pose-Free Novel View Synthesis (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.