Research map / 3D vision / Human pose estimation
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FoundationPose unifies 6D object pose estimation and tracking for novel objects in model-based (CAD) and model-free (few reference images) setups, using LLM-aided synthetic training, a transformer refiner and ranker, and an object-centric neural SDF for novel-view RGBD rendering.
Problem
Instant 6D pose estimation and tracking of arbitrary novel objects from RGBD input, where prior methods are specialized to either model-based or model-free setups and to either estimation or tracking, and often require per-instance training.
Approach
A single render-and-compare framework: an LLM-aided, diffusion-textured synthetic data pipeline for large-scale training, an object-centric neural SDF to synthesize RGBD views when no CAD model exists, transformer-based pose refinement, and a hierarchical transformer pose ranker trained with a contrastive pose-conditioned triplet loss.
Key results
- Model-free on YCB-Video: mean ADD-S AUC 97.4 and ADD AUC 91.5 without fine-tuning, versus 88.4/42.1 for FS6D (which requires fine-tuning).
- Model-based on BOP core datasets: mean AR 83.3 versus 58.6 for MegaPose-RGBD and 79.7 for SurfEmb+ICP (instance-level).
- Model-free LINEMOD ADD-0.1d average 99.9 versus 91.5 for FS6D+ICP; tracking on YCBInEOAT reaches mean ADD-S 96.40 and ADD 93.09, outperforming the instance-level se(3)-TrackNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FoundationPose instead of FS6D | Model-free pose estimation · YCB-Video | AUC of ADD: 42.1 -> 91.5 | Helped |
| use FoundationPose instead of FS6D | Model-free pose estimation · YCB-Video | AUC of ADD-S: 88.4 -> 97.4 | Helped |
| use FoundationPose instead of FS6D + ICP | Model-free pose estimation · LINEMOD | ADD-0.1d: 91.5 -> 99.9 | Helped |
| use FoundationPose instead of the strongest listed baseline | Model-based pose estimation · Occluded-LINEMOD | AR: 75.8 -> 78.8 | Helped |
| use FoundationPose instead of the strongest listed baseline | Model-based pose estimation · T-LESS | AR: 82.8 -> 83.0 | Helped |
| use FoundationPose instead of the strongest listed baseline | Model-based pose estimation · YCB-Video | AR: 80.6 -> 88.0 | Helped |
| use FoundationPose instead of se(3)-TrackNet | Pose tracking · YCBInEOAT | AUC of ADD: 92.66 -> 93.09 | Helped |
| use FoundationPose instead of se(3)-TrackNet | Pose tracking · YCBInEOAT | AUC of ADD-S: 95.53 -> 96.42 | Helped |
| use FoundationPose instead of se(3)-TrackNet | Pose tracking · YCB-Video | AUC of ADD: 93.0 -> 96.0 | Helped |
| use FoundationPose instead of se(3)-TrackNet | Pose tracking · YCB-Video | AUC of ADD-S: 95.7 -> 97.9 | Helped |
| add LLM texture augmentation (ablation) | FoundationPose · YCB-Video | AUC of ADD: 90.83 -> 91.52 | Helped |
| add LLM texture augmentation (ablation) | FoundationPose · YCB-Video | AUC of ADD-S: 97.38 -> 97.40 | No clear effect |
Surprising findings
- Hierarchical comparison across all pose hypotheses outperforms an InfoNCE-based contrastive loss (ADD 91.52 vs 89.39 on YCB-Video ablation).
- Using only about 4 reference images still outperforms FS6D with 16 reference images on YCB-Video.
Limitations
- Pose estimation takes about 1.3 s per object, so it is not real-time without switching to tracking mode.
- Requires RGBD input and an off-the-shelf detector (Mask R-CNN or CNOS) for initialization.
- Evaluated only on rigid single objects; state estimation beyond single rigid objects is left to future work.
Experimental setup
Datasets: Objaverse-LVIS GSO ImageNet MS-COCO LINEMOD Occluded-LINEMOD YCB-Video T-LESS YCBInEOAT
Benchmarks: LINEMOD Occluded-LINEMOD YCB-Video T-LESS YCBInEOAT
Metrics: AUC of ADD AUC of ADD-S ADD-0.1d recall BOP average recall (AR)
Baselines: FS6D OnePose++ LatentFusion PREDATOR LoFTR MegaPose-RGBD se(3)-TrackNet ICG PoseRBPF DeepIM
Builds on: NeRF FS6D MegaPose NeuS-style SDF rendering TexFusion
Closest papers by meaning
- Vision Foundation Model Enables Generalizable Object Pose Estimation (NeurIPS 2024 · similarity 0.89)
- Pos3R: 6D Pose Estimation for Unseen Objects Made Easy (CVPR 2025 · similarity 0.88)
- 6D Object Pose Tracking in Internet Videos for Robotic Manipulation (ICLR 2025 · similarity 0.88)
- Open-Vocabulary Object 6D Pose Estimation (CVPR 2024 · similarity 0.88)
- SingRef6D: Monocular Novel Object Pose Estimation with a Single RGB Reference (NeurIPS 2025 · similarity 0.87)
- Any6D: Model-free 6D Pose Estimation of Novel Objects (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.