Research map / 3D vision / Human pose estimation
6D Object Pose Tracking in Internet Videos for Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Estimates 6D object poses of arbitrary objects via retrieved CAD models (DINOv2 foreground feature averaging), LLM-based metric scale grounding, and tracking with CoTracker+PnP, then retargets Internet-video object trajectories to a Franka robot via trajectory optimization.
Problem
Extracting temporally consistent 6D object pose trajectories of manipulated objects from uncontrolled Internet instructional videos where the exact object mesh is unknown, and transferring them to robots.
Approach
Training-free pipeline that retrieves a similar CAD model from a ~50k-object database via DINOv2 foreground feature averaging, aligns it by patch-feature matching over rotations, grounds absolute scale using monocular depth plus GPT-4-generated object size priors matched with CLIP, and refines trajectories with dense point tracking and PnP.
Key results
- On YCB-V the method reaches AR 49.86 versus 42.95 for FoundPose and 25.76 for MegaPose, and on HOPE-Video AR 45.98 versus 42.30 for FoundPose.
- Replacing OpenShape retrieval with the proposed FFA retrieval raises YCB-V AR from 34.51 to 49.86.
- LLM-based scale estimation improves AR_CH from 11.89 (constant 10cm) to 18.53 on YCB-V.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed method instead of FoundPose | 6D pose estimation · YCB-V | AR: 42.95 -> 49.86 | Helped |
| Use the proposed method instead of FoundPose | 6D pose estimation · HOPE-Video | AR: 42.30 -> 45.98 | Helped |
| Refine object poses with tracked 3D-2D correspondences | Ours · In-the-wild videos | Relative rotation: 1.719 -> 1.205 | Helped |
| Refine object poses with tracked 3D-2D correspondences | Ours · In-the-wild videos | Relative projected translation: 0.091 -> 0.086 | Helped |
| Replace FFA retrieval with OpenShape (ablation) | Ours · YCB-V | AR: 49.86 -> 34.51 | Hurt |
| Use FFA instead of the per-view averaged CLS token descriptor (ablation) | Ours · YCB-V | AR: 45.05 -> 49.86 | Helped |
| Use LLM-based scale estimation instead of constant 10cm scale (ablation) | Ours · YCB-V | AR_CH: 11.89 -> 18.53 | Helped |
Surprising findings
- The depth error metric was not informative: a zero-depth-velocity baseline outperforms all compared methods (0.0289).
- Retrieved CAD models are used without ground-truth meshes, and the method still outperforms methods that use onboarding or known-mesh pipelines on YCB-V and HOPE-Video.
Limitations
- Fails when segmentation quality is low or the object is significantly occluded, so retrieval and scale estimation may fail.
- Retrieval selects semantically similar objects that may not be geometrically similar, causing pose errors.
- Assumes a static camera for tracking; moving cameras need external COLMAP-based correction that is not implemented.
Experimental setup
Datasets: Objaverse-LVIS Google Scanned Objects YCB-V HOPE-Video EPIC-KITCHENS
Benchmarks: YCB-V HOPE-Video in-the-wild instructional videos (32 videos, 6947 frames) EPIC-KITCHENS
Metrics: AR AR_CoU AR_CH AR_pCH relative rotation error relative projected translation error relative depth error
Baselines: MegaPose GigaPose FoundPose OpenShape
Builds on: FoundPose CNOS MegaPose CoTracker SAM 2
Closest papers by meaning
- UA-Pose: Uncertainty-Aware 6D Object Pose Estimation and Online Object Completion with Partial References (CVPR 2025 · similarity 0.88)
- FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects (CVPR 2024 · similarity 0.88)
- ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation (NeurIPS 2026 · similarity 0.87)
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.87)
- CAP-Net: A Unified Network for 6D Pose and Size Estimation of Categorical Articulated Parts from a Single RGB-D Image (CVPR 2025 · similarity 0.87)
- KPA-Tracker: Towards Robust and Real-Time Category-Level Articulated Object 6D Pose Tracking (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.