Research map / 3D vision / Human pose estimation

ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation

NeurIPS 2026 · 3D vision › Human pose estimation · approach: Instance-level 6D object pose · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Recasts monocular 6-DoF pose tracking as video-to-video translation: a LoRA-fine-tuned Wan-14B video diffusion model turns a source video plus one marked pixel into a synthetic cube "proxy video", whose pose is recovered with PnP and smoothing.

Problem

Tracking 6-DoF pose of arbitrary objects and surfaces from monocular video typically requires 3D models, depth, masks, or task-specific features, and fails on textureless, transparent, reflective, or deformable surfaces.

Approach

Fine-tune a pretrained video diffusion model to translate the input video into a proxy video of a colored cube undergoing the same local rigid motion as the marked pixel's surface; the cube's known geometry and appearance let classical PnP recover the pose, with no object-specific inputs.

Key results

• On HO3D, ProxyPose (one prompt) reaches ATE 15.79 mm and ARE 5.13 deg, versus 25.44 mm and 15.38 deg for FoundationPose (track) on the same sequences.
• On YCBInEOAT, ProxyPose (one prompt) achieves ATE 30.07 mm and ARE 15.07 deg, the best mean ATE and ARE among the methods compared.
• Ablation: removing the noise schedule offset raises HO3D ATE from 81.65 mm to 507.5 mm on the synthetic set, showing the first-frame anchor is critical.

Surprising findings

• Training on only 300 synthetic sequences already achieves performance comparable to the 35k-sequence model on the synthetic benchmark.
• ProxyPose (one query) operates from monocular RGB alone, yet beats baselines that use CAD models or ground-truth depth on several metrics.

Limitations

• Inference takes about 5.5 minutes per video on one H100, far from real time.
• Fast motion causes VAE blur that degrades contour detection; drift occurs for textureless or reflective objects and fluids.
• Frame count is bounded by the video model's generation length, and only relative pose is recovered.

Experimental setup

Datasets: Trellis-500K (Objaverse assets) HO3D YCBInEOAT

Benchmarks: HO3D YCBInEOAT held-out synthetic dataset

Metrics: ATE ARE RPE-t RPE-r 2D-dist T-Drift ADD ADD-S

Baselines: FoundationPose (track) FoundationPose (registration) Any6D One2Any Oryon ConceptPose BundleSDF SpatialTracker V2 CoTracker 3 + Depth Anything 3

Builds on: Wan flow matching LoRA Point Prompting CoTracker 3 FoundationPose

Closest papers by meaning

• 6D Object Pose Tracking in Internet Videos for Robotic Manipulation (ICLR 2025 · similarity 0.87)
• FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects (CVPR 2024 · similarity 0.87)
• ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.86)
• Dynamic Camera Poses and Where to Find Them (CVPR 2025 · similarity 0.86)
• PoseTraj: Pose-Aware Trajectory Control in Video Diffusion (CVPR 2025 · similarity 0.86)
• TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.86)