Research map / 3D vision / 3D generation
Unsupervised 3D Structure Inference from Category-Specific Image Collections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the first unsupervised 3D keypoint inference method that learns sparse 3D keypoints and a shape-space model from single-view category image collections by extending AutoLink with a 3D shape basis, reprojection and repulsion losses.
Problem
Inferring 3D keypoints of general object categories from inhomogeneous single-view image collections without 2D/3D keypoint annotations, multi-view input, videos, or shape priors.
Approach
Jointly train a 2D keypoint heatmap encoder (AutoLink) with a globally learned linear 3D shape space model (mean shape plus basis) whose input-dependent coefficients are predicted by an MLP, coupled to 2D keypoints via a reprojection loss under an orthographic camera, plus a repulsion loss to prevent keypoint collapse.
Key results
- On CelebA-Wild 2D keypoints with K=8, the method reaches 5.21% NME versus 5.39% for AutoLink, and it leads at K=16/24/32 (3.97%, 3.54%, 3.48%).
- On AFLW2000-3D 3D keypoint inference (trained on 300W-LP), the method achieves 8.48% NME, better than AutoLink+MiDaS (9.23%) and AutoLink+Unsup3d (11.47%), though behind supervised 3DDFA (4.94%).
- Ablation shows removing the repulsion loss causes keypoints to collapse into a degenerate configuration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed 3D shape-space method instead of AutoLink for 2D keypoint inference | CELEBA WILD, K=8 · CELEBA WILD | NME: 5.39% -> 5.21% | Helped |
| Use the proposed method instead of AutoLink for 2D keypoint inference | CELEBA WILD, K=16 · CELEBA WILD | NME: 4.69% -> 3.97% | Helped |
| Use the proposed method instead of AutoLink for 2D keypoint inference | CELEBA WILD, K=24 · CELEBA WILD | NME: 3.99% -> 3.54% | Helped |
| Use the proposed method instead of AutoLink for 2D keypoint inference | CELEBA WILD, K=32 · CELEBA WILD | NME: 3.77% -> 3.48% | Helped |
| Replace AutoLink + MiDaS with the proposed 3D shape-space method | 3D keypoint inference trained on 300W-LP · AFLW2000-3D | NME: 9.23% -> 8.48% | Helped |
| Replace AutoLink + Unsup3d with the proposed 3D shape-space method | 3D keypoint inference trained on 300W-LP · AFLW2000-3D | NME: 11.47% -> 8.48% | Helped |
| Use the proposed unsupervised method instead of supervised 3DDFA | 3D keypoint inference trained on 300W-LP · AFLW2000-3D | NME: 4.94% -> 8.48% | Hurt |
| Increase the number of keypoints from 16 to 32 (ablation) | Proposed method · CELEBA WILD | captured structure detail | Helped |
| Add the repulsion loss Lrep (ablation) | Proposed method · CELEBA WILD | keypoint structure quality | Helped |
Surprising findings
- The method fails to recover correct poses on HUMAN3.6M with strong articulation, producing incorrect instance-specific deformations.
Limitations
- Fails on datasets with large pose variations or strong articulation (e.g., HUMAN3.6M) without domain knowledge.
- Only evaluated quantitatively for 3D on faces (AFLW2000-3D); no other 3D baseline exists in the unsupervised setting.
- Relies on moderate intra-class shape variation and aligned or moderately varying poses.
Experimental setup
Datasets: CelebA-Wild CUB-200-2011 300W-LP AFHQ HORSE (CycleGAN dataset)
Benchmarks: CelebA-Wild AFLW2000-3D CUB-200-2011 HORSE AFHQ
Metrics: normalized mean L2 error (NME)
Baselines: AutoLink AutoLink + MiDaS AutoLink + Unsup3d 3DDFA LatentKeypointGAN IMM Lorenz et al. SCOPS GANSeg
Builds on: AutoLink Thewlis et al. unsupervised keypoints Unsup3d MiDaS VGG
Closest papers by meaning
- KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control (CVPR 2021 · similarity 0.90)
- Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features (NeurIPS 2024 · similarity 0.88)
- Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Features (CVPR 2024 · similarity 0.88)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.87)
- ESCAPE: Encoding Super-keypoints for Category-Agnostic Pose Estimation (CVPR 2024 · similarity 0.87)
- KeyPose: Category-Level 6D Object Pose Estimation with Self-Adaptive Keypoints (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.