Research map / 3D vision / Human pose estimation
Vision Foundation Model Enables Generalizable Object Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VFM-6D, which freezes DINOv2 and trains a 2D-to-3D feature lifting module and a shape-matching module on synthetic Blender data to perform two-stage (viewpoint then NOCS coordinate-map) category-level and instance-level object pose estimation for unseen categories.
Problem
Object pose estimation generalizes poorly: instance-level methods need CAD models or per-instance reference images, and category-level methods only work for categories seen in training.
Approach
Use a frozen vision foundation model (DINOv2) with category-level multi-view reference images; lift 2D foundation features to 3D using point-cloud position embeddings for view-aware viewpoint matching, and fuse foundation image semantics into point-cloud shape encoding for NOCS coordinate-map estimation, trained only on cheap synthetic data.
Key results
- On Wild6D's 5 novel categories, VFM-6D reaches 19.3% at 5°2cm and 44.2% at 10°5cm, surpassing category-agnostic PoseContrast and ZSP.
- On 20 unseen CO3D categories, VFM-6D averages 50.2 Acc.15° / 67.4 Acc.30° versus 27.1 / 44.7 for ZSP.
- On LINEMOD (instance-level), VFM-6D averages 90.3 ADD-0.1d, beating zero-shot LatentFusion (87.1) and OSOP (82.0) and comparable to few-shot FS6D (91.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VFM-6D instead of category-agnostic baselines | Wild6D novel-category pose estimation · Wild6D (5 novel categories) | 5°2cm: 20.1 -> 19.3 | No clear effect |
| use VFM-6D instead of category-agnostic baselines | Wild6D novel-category pose estimation · Wild6D (5 novel categories) | 10°2cm: 29.0 -> 34.9 | Helped |
| use VFM-6D instead of ZSP | CO3D category-level pose estimation · CO3D (20 unseen categories) | Acc.15°: 27.1 -> 50.2 | Helped |
| use VFM-6D instead of ZSP | CO3D category-level pose estimation · CO3D (20 unseen categories) | Acc.30°: 44.7 -> 67.4 | Helped |
| use VFM-6D instead of FS6D | LINEMOD instance-level pose estimation · LINEMOD | ADD-0.1d: 91.5 -> 90.3 | Hurt |
| add the 2D-to-3D feature lifting module (ablation) | VFM-6D · CO3D (5-category subset) | Acc.15°: 48.8 -> 58.4 | Helped |
| add the 2D-to-3D feature lifting module (ablation) | VFM-6D · CO3D (5-category subset) | Acc.30°: 62.9 -> 73.8 | Helped |
| add the foundation feature-based shape representation module (ablation) | VFM-6D · CO3D (5-category subset) | Acc.15°: 48.6 -> 58.4 | Helped |
| add the foundation feature-based shape representation module (ablation) | VFM-6D · CO3D (5-category subset) | Acc.30°: 66.7 -> 73.8 | Helped |
| add both feature lifting and the shape representation module (ablation) | VFM-6D · CO3D (5-category subset) | Acc.15°: 35.2 -> 58.4 | Helped |
| add both feature lifting and the shape representation module (ablation) | VFM-6D · CO3D (5-category subset) | Acc.30°: 53.0 -> 73.8 | Helped |
Surprising findings
- Foundation-feature template matching alone (DINO-v1 and DINOv2) gives nearly indistinguishable cosine similarities across views, motivating the lifting module.
- Training from scratch with a point cloud transformer generalizes better than from-scratch training without the foundation model, though the foundation model still does best.
Limitations
- Degrades under severe occlusion (>60% occlusion rate), since occlusion affects foundation features.
- Requires RGB-D input (depth), limiting use on internet-scale RGB-only data.
- No statistical significance / error bars reported.
Experimental setup
Datasets: ShapeNet NOCS Wild6D CO3D LINEMOD
Benchmarks: Wild6D CO3D LINEMOD LINEMOD-Occlusion
Metrics: mAP at a°b cm Acc.15° Acc.30° ADD-0.1d
Baselines: SPD SGPA DualPoseNet GPV-Pose PoseContrast ZSP LoFTR LightGlue GeDi LatentFusion OSOP FS6D
Builds on: DINOv2 NOCS PoseContrast Gen6D ZSP
Closest papers by meaning
- FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects (CVPR 2024 · similarity 0.89)
- GenCOPE: Syn2Real Generalized Category-Level Object Pose Estimation for Robotic Picking (NeurIPS 2026 · similarity 0.88)
- SchemaPose: RGB-Based Category-Level Pose Estimation with Parametric Category Schema (NeurIPS 2026 · similarity 0.87)
- Universal Features Guided Zero-Shot Category-Level Object Pose Estimation (AAAI 2025 · similarity 0.87)
- Instance-Adaptive and Geometric-Aware Keypoint Learning for Category-Level 6D Object Pose Estimation (CVPR 2024 · similarity 0.87)
- 3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.