Research map / 3D vision / Human pose estimation
Can Generative Video Models Help Pose Estimation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses off-the-shelf generative video models to hallucinate frames between two low-overlap images, then selects the most self-consistent pose estimate (medoid distance) from DUSt3R/MASt3R runs on the augmented frame sets.
Problem
Relative camera pose estimation between two images with little or no overlap, where feature correspondences are unavailable and existing pose estimators struggle.
Approach
Treat a pretrained video interpolation model as a world prior that fills in a dense visual transition between two images, then feed generated frames to a pose estimator and pick the output with a self-consistency (medoid distance) score across sampled frame subsets, videos, and generation directions.
Key results
- On Cambridge Landmarks with DUSt3R, mean rotation error drops from 13.28 to 10.78 degrees using Runway generated frames.
- On ScanNet with DUSt3R and Dream Machine frames, mean rotation and translation errors fall from 21.31/24.72 to 17.65/15.88 degrees.
- On DL3DV-10K with DUSt3R and Dream Machine, translation accuracy at 30 degrees rises from 89% to 94.67%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Runway-generated frames to DUSt3R and select using self-consistency | DUSt3R · Cambridge Landmarks | MRE: 13.28° to 10.78° | Helped |
| Add Dream Machine-generated frames to DUSt3R and select using self-consistency | DUSt3R · ScanNet | MRE: 21.31° to 17.65° | Helped |
| Add Dream Machine-generated frames to DUSt3R and select using self-consistency | DUSt3R · ScanNet | MTE: 24.74° to 15.88° | Helped |
| Add Dream Machine-generated frames to MASt3R and select using self-consistency | MASt3R · Cambridge Landmarks | MRE: 36.55° to 27.47° | Helped |
| Add Dream Machine-generated frames to MASt3R and select using self-consistency | MASt3R · ScanNet | AUC30: 55.10% to 58.28% | Helped |
| Add Dream Machine-generated frames to DUSt3R and select using self-consistency | DUSt3R · DL3DV-10K | MTE: 13.08° to 8.72° | Helped |
| Add Dream Machine-generated frames to DUSt3R and select using self-consistency | DUSt3R · DL3DV-10K | tacc@30°: 89% to 94.67% | Helped |
| Add video-generated frames to DUSt3R and select using self-consistency | DUSt3R · NAVI | MRE: 8.65° to 7.85° | Helped |
| Add video-generated frames to DUSt3R and select using self-consistency | DUSt3R · NAVI | MTE: 7.88° to 6.24° | Helped |
| Add Runway-generated frames to MASt3R and select using self-consistency | MASt3R · NAVI | MRE: 5.59° to 5.28° | Helped |
| Add Runway-generated frames to MASt3R and select using self-consistency | MASt3R · NAVI | MTE: 5.23° to 5.20° | Helped |
Surprising findings
- Video models show a left-to-right panning bias regardless of interpolation direction, so the authors flip input order to mitigate it.
- Gains are smaller on center-facing datasets where input views already overlap substantially.
Limitations
- Video models are expensive and slow, limiting evaluation scope.
- Generated videos lack multi-view consistency and sometimes all samples are low quality.
- Results are sensitive to prompts, camera intrinsics, and aspect ratio; gains are modest on center-facing datasets.
Experimental setup
Datasets: Cambridge Landmarks ScanNet DL3DV-10K NAVI
Benchmarks: Cambridge Landmarks ScanNet DL3DV-10K NAVI
Metrics: mean rotation error mean translation error rotation accuracy at 5/15/30 degrees translation accuracy at 5/15/30 degrees AUC30
Baselines: SIFT+nearest neighbors LoFTR DUSt3R MASt3R
Builds on: DUSt3R MASt3R DynamiCrafter
Closest papers by meaning
- ExPose: Reinforcing Video Generation Models for Extreme Pose Estimation (CVPR 2026 · similarity 0.90)
- PoseCrafter: Extreme Pose Estimation with Hybrid Video Synthesis (NeurIPS 2025 · similarity 0.88)
- AdaptPose: Cross-Dataset Adaptation for 3D Human Pose Estimation by Learnable Motion Generation (CVPR 2022 · similarity 0.88)
- Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis (NeurIPS 2024 · similarity 0.88)
- Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos (CVPR 2024 · similarity 0.87)
- PoseIRM: Enhance 3D Human Pose Estimation on Unseen Camera Settings via Invariant Risk Minimization (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.