Research map / 3D vision / Human pose estimation
Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns category-level 3D pose estimation from casually captured object-centric videos without human labels by aligning videos via a cyclic 3D geometry-plus-DINOv2-feature distance and training a neural-mesh correspondence model.
Problem
Category-level 3D object pose estimation normally needs human pose labels, CAD models, or RGB-D input; the paper targets learning it unsupervised from casual object-centric videos and predicting pose from a single image.
Approach
A weighted 3D cyclical-distance alignment of coarse video reconstructions with self-supervised DINOv2 surface features aligns videos into a canonical frame; the aligned meshes then supervise a vMF-based neural mesh correspondence model with render-and-compare pose fitting.
Key results
- Unsupervised alignment on CO3D reaches 77.0% mean 30° accuracy over 20 categories, improving over UCD+ (69.8%) and ZSP (49.4%).
- On PASCAL3D+ (7 categories) the unsupervised method achieves 69.2% mean 30° accuracy versus 46.0% for ZSP, which uses depth and pose annotations.
- On ObjectNet3D the method reaches 52.4% 30° accuracy for regression-free render-and-compare vs. 44.5% for direct regression, and runs in 0.22 s per sample versus 10.92 s for ZSP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed weighted 3D alignment instead of UCD+ | CO3D unsupervised alignment · CO3D, 20 categories | Acc. 30°: 69.8% to 77.0% | Helped |
| use the proposed in-the-wild pose estimator instead of ZSP | Category-level pose estimation · PASCAL3D+, 7 categories | 30° accuracy: 46.0 to 69.2 | Helped |
| use the proposed in-the-wild pose estimator instead of ZSP | Category-level pose estimation · ObjectNet3D, 20 categories | 30° accuracy: 42.2 to 52.4 | Helped |
| replace render-and-compare pose inference with neural network regression (ablation) | Ours · PASCAL3D+ | 30° accuracy: 69.2 to 66.0 | Hurt |
| replace render-and-compare pose inference with neural network regression (ablation) | Ours · PASCAL3D+ | 15° accuracy: 41.3 to 25.8 | Hurt |
| replace render-and-compare pose inference with neural network regression (ablation) | Ours · ObjectNet3D | 30° accuracy: 52.4 to 44.5 | Hurt |
| replace render-and-compare pose inference with neural network regression (ablation) | Ours · ObjectNet3D | 15° accuracy: 25.5 to 16.4 | Hurt |
| tune appearance weight α and cyclical-distance temperature τ (best setting α = 0.2 and τ = 100) (ablation) | Unsupervised alignment · CO3D alignment | 30° accuracy: 75.2% | Helped |
| use render-and-compare pose inference instead of neural network regression | Ours · ObjectNet3D | seconds per sample: 10.92 seconds to 0.22 seconds | Helped |
Surprising findings
- Cross-category generalization from a toybus in CO3D to real buses in PASCAL3D+ works despite the domain gap.
- Euclidean correspondences are weighted about four times as much as feature correspondences in the optimal alignment setting.
Limitations
- Does not yet reach the performance of fully supervised baselines.
- Assumes rigid shape models; deformable objects are not handled.
- Learns from pre-recorded videos rather than continuous streams.
Experimental setup
Datasets: CO3D PASCAL3D+ ObjectNet3D
Benchmarks: CO3D PASCAL3D+ ObjectNet3D
Metrics: 30-degree accuracy 15-degree accuracy
Baselines: ZSP UCD+ TEASER++ StarMap VoGE Regression (6D rotation)
Builds on: DINOv2 Neural Mesh Models (NeMo) ZSP (Goodwin et al. zero-shot pose) Umeyama alignment RANSAC
Closest papers by meaning
- Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation (ICLR 2024 · similarity 0.91)
- Emergence of a Shared Canonical Object Frame from In-the-Wild Videos (NeurIPS 2026 · similarity 0.90)
- NOPE: Novel Object Pose Estimation from a Single Image (CVPR 2024 · similarity 0.89)
- SchemaPose: RGB-Based Category-Level Pose Estimation with Parametric Category Schema (NeurIPS 2026 · similarity 0.89)
- Universal Features Guided Zero-Shot Category-Level Object Pose Estimation (AAAI 2025 · similarity 0.88)
- Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.