Research map / 3D vision / Face and shape
Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Common3D learns category-specific 3D morphable models of common objects from object-centric videos in a self-supervised way, representing appearance as neural features and training a DINOv2 adapter with a contrastive correspondence objective.
Problem
3D morphable models exist only for a few categories because they need costly 3D data acquisition and category-specific training; learning 3D object priors for common objects from in-the-wild images is hard.
Approach
Represent a category as a learned DMTet-based 3D template with an image-conditioned deformation field and neural feature appearance (instead of RGB), and train a DINOv2 adapter jointly with a dense contrastive surface-probability objective that exploits mesh vertex correspondences.
Key results
- On PASCAL3D+ 7 categories, pose accuracy (30°) averages 75.3% vs 69.2% for UOP3D (unsupervised).
- On ObjectNet3D 20 categories, 30° pose accuracy improves from 52.4% to 56.8%.
- On SPair-71k semantic correspondence, PCK@0.1 rises from 47.9% (DINOv2) to 54.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Common3D instead of UOP3D | 3D pose estimation · PASCAL3D+ | 30° accuracy: 69.2% to 75.3% | Helped |
| use Common3D instead of UOP3D | 3D pose estimation · ObjectNet3D | 30° accuracy: 52.4% to 56.8% | Helped |
| use Common3D instead of UOP3D | instance segmentation · PASCAL3D+ | IoU: 60.3% to 64.1% | Helped |
| use Common3D instead of UOP3D | instance segmentation · ObjectNet3D | IoU: 64% to 67% | Helped |
| use the Common3D feature adapter with DINOv2 features | semantic correspondence estimation · SPair-71k | PCK@0.1: 47.9% to 54% | Helped |
| add the feature adapter (ablation) | Common3D · PASCAL3D+ | 30° accuracy: 25.9 -> 75.3 | Helped |
| add the dense appearance loss (ablation) | Common3D · PASCAL3D+ | IoU: 58.8 -> 64.1 | Helped |
| add instance-level deformation (ablation) | Common3D · PASCAL3D+ | IoU: 58.6 -> 64.1 | Helped |
| remove the feature adapter and use reconstruction loss (ablation) | Common3D · PASCAL3D+ | 30° accuracy: 75.3 -> 13.9 | Hurt |
| remove instance-level deformation (ablation) | Common3D · PASCAL3D+ | 30° accuracy: 75.8 vs 75.3 | No clear effect |
Surprising findings
- Instance-level deformation has only marginal effect on coarse pose estimation but is essential for segmentation and correspondence.
Limitations
- Domain gap between CO3D training videos and in-the-wild test data limits correspondence performance.
- Performance lags on categories with few shape variations such as cellphone, microwave, and toaster.
Experimental setup
Datasets: CO3D PASCAL3D+ ObjectNet3D SPair-71k
Benchmarks: PASCAL3D+ ObjectNet3D SPair-71k
Metrics: 30-degree pose accuracy 10-degree pose accuracy IoU PCK@0.1
Baselines: StarMap VoGE ZSP UOP3D SpherMaps DINOv2
Builds on: DMTet DINOv2 UOP3D VoGE SpherMaps
Closest papers by meaning
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.89)
- Learning Feature Aggregation for Deep 3D Morphable Models (CVPR 2021 · similarity 0.89)
- Regressing Robust and Discriminative 3D Morphable Models With a Very Deep Neural Network (CVPR 2017 · similarity 0.89)
- Combining 3D Morphable Models: A Large Scale Face-And-Head Model (CVPR 2019 · similarity 0.88)
- Learning Category-Specific 3D Shape Models From Weakly Labeled 2D Images (CVPR 2017 · similarity 0.88)
- Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.