Research map / 3D vision / Human pose estimation
Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Key-Grid is an unsupervised 3D keypoint detector that builds a grid heatmap from predicted keypoint skeletons to guide autoencoder reconstruction, targeting both rigid and deformable objects.
Problem
Unsupervised 3D keypoint detection with semantic consistency fails on deformable objects (e.g., garments), where keypoints drift under large shape variations.
Approach
Embed predicted keypoints into a dense 3D grid heatmap whose values are max-weighted distances from grid points to the keypoint "skeleton" segments, and feed this heatmap plus multi-layer encoder features into a PointNet++-style decoder, trained with Chamfer reconstruction and farthest-point keypoint losses.
Key results
- On ShapeNetCoreV2 (10 keypoints), Key-Grid averages 72.8 DAS vs 69.7 for SC3K and 49.0 mIoU vs 45.6 for SM.
- On ClothesNet drop deformation, Key-Grid averages 89.8 DAS vs 81.8 for KeypointDeformer; on folded pants Key-Grid reaches 100.0 DAS vs 72.4 for KD.
- Ablation removing the farthest point keypoint loss drops DAS to 15.6 on airplane and 17.9 on folded shirt, the largest ablation drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Key-Grid instead of the strongest listed baseline | Key-Grid · ShapeNetCoreV2 average (13 categories) | DAS: 69.7 -> 72.8 | Helped |
| use Key-Grid instead of the strongest listed baseline | Key-Grid · ShapeNetCoreV2 average (13 categories) | mIoU: 45.6 -> 49.0 | Helped |
| use Key-Grid instead of the strongest listed baseline | Key-Grid · ClothesNet dropping average (13 categories) | DAS: 81.8 -> 89.8 | Helped |
| use Key-Grid instead of the strongest listed baseline | Key-Grid · ClothesNet dragging average (13 categories) | DAS: 58.6 -> 67.7 | Helped |
| use Key-Grid instead of PointConv | Key-Grid · KeypointNet average (Airplane, Chair, Car) | mIoU: 87.0 -> 75.1 | Hurt |
| add encoder information to the decoder (ablation) | Key-Grid · ShapeNetCoreV2 Airplane and Chair; ClothesNet folded shirt and pant | DAS: 73.4 -> 84.7; 82.6 -> 93.4; 81.7 -> 92.0; 90.8 -> 100.0 | Helped |
| add the grid heatmap to the decoder (ablation) | Key-Grid · ShapeNetCoreV2 Airplane and Chair; ClothesNet folded shirt and pant | DAS: 73.2 -> 84.7; 81.7 -> 93.4; 79.5 -> 92.0; 87.5 -> 100.0 | Helped |
| add the similarity loss (ablation) | Key-Grid · ShapeNetCoreV2 Airplane and Chair; ClothesNet folded shirt and pant | DAS: 71.5 -> 84.7; 78.6 -> 93.4; 77.7 -> 92.0; 84.5 -> 100.0 | Helped |
| add the farthest point keypoint loss (ablation) | Key-Grid · ShapeNetCoreV2 Airplane and Chair; ClothesNet folded shirt and pant | DAS: 15.6 -> 84.7; 13.7 -> 93.4; 17.9 -> 92.0; 17.0 -> 100.0 | Helped |
Surprising findings
- Using a SPRIN SE(3)-invariant backbone directly in place of PointNet++ made Key-Grid training fail to converge, requiring a teacher-student USEEK setup instead.
- On ShapeNetCoreV2 the method is not best on all categories; Skeleton Merger outperforms Key-Grid on mIoU for airplane, vessel, knife and guitar.
Limitations
- The number of keypoints must be set manually in advance.
- Memory use is higher than Skeleton Merger.
- Mixed mIoU results: Skeleton Merger is better on several rigid categories with straight-line structures.
Experimental setup
Datasets: ShapeNetCoreV2 KeypointNet ClothesNet Deep Fashion3D V2 SUN3D
Benchmarks: ShapeNetCoreV2 KeypointNet ClothesNet Deep Fashion3D V2 SUN3D
Metrics: DAS mIoU
Baselines: KeypointDeformer Skeleton Merger SC3K PointNet SpiderCNN PointConv USEEK
Builds on: Skeleton Merger PointNet++ USEEK USIP KeypointDeformer
Closest papers by meaning
- KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control (CVPR 2021 · similarity 0.90)
- TUSK: Task-Agnostic Unsupervised Keypoints (NeurIPS 2022 · similarity 0.89)
- Unsupervised 3D Structure Inference from Category-Specific Image Collections (CVPR 2024 · similarity 0.88)
- Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Features (CVPR 2024 · similarity 0.87)
- KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation (CVPR 2024 · similarity 0.86)
- KeyPose: Category-Level 6D Object Pose Estimation with Self-Adaptive Keypoints (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.