Research map / 3D vision / 3D generation
Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Back-projects features from frozen 2D foundation models (mainly DINO ViT) onto 3D meshes and uses a distribution-matching keypoint candidate optimization to perform few-shot 3D keypoint detection on KeypointNet.
Problem
Few-shot 3D keypoint detection on meshes requires both semantic and geometric awareness with high localization accuracy, which prior 3D-pretrained or axiomatic descriptors handle poorly, and symmetric shapes cause keypoint collapse.
Approach
Render shapes from many viewpoints, extract dense features with a frozen pre-trained 2D encoder (DINO), back-project and average them onto the surface with Gaussian geodesic reweighting, then select keypoint candidates by optimizing feature and relative geodesic-distance matching so the keypoint distribution matches the few-shot examples.
Key results
- On KeypointNet, B2-3D improves over the previous best FSKD by 93% mean relative IoU across distance thresholds, with over 200% improvement at threshold 0.02.
- Part segmentation transfer with back-projected DINO features reaches 71.0% IoU on ShapeNet part, versus 69.2% for NCP.
- Features from the smaller DINO ViT-S perform almost equally well to ViT-G, while textured ShapeNet meshes slightly degrade performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use B2-3D with back-projected DINO features instead of FSKD | few-shot keypoint detection · KeypointNet (airplane, chair, table) | IoU: 93% mean relative improvement; over 200% improvement at a distance threshold of 0.02 | Helped |
| replace the DINO ViT-G features with distilled ViT-S features (ablation) | B2-3D · KeypointNet (airplane, chair, table) | IoU | No clear effect |
| use ShapeNet mesh textures during feature extraction (ablation) | B2-3D · KeypointNet (airplane, chair, table) | IoU | Hurt |
| replace the keypoint distribution optimization with nearest-neighbor keypoint selection (ablation) | B2-3D · KeypointNet (airplane, chair, table) | IoU | Hurt |
| retrieve the closest labeled shape before optimization (ablation) | B2-3D · KeypointNet (airplane, chair, table) | IoU | No clear effect |
| use one labeled sample instead of three (ablation) | B2-3D · KeypointNet (airplane, chair, table) | IoU | Hurt |
| replace DINO features with CLIP, SAM, or EfficientNet features (ablation) | B2-3D keypoint optimization · KeypointNet (airplane, chair, table) | IoU | Hurt |
| use back-projected DINO features and nearest-neighbor classification for part-label transfer | part segmentation transfer · ShapeNet part dataset | IoU: 69.2% -> 71.0% | Helped |
Surprising findings
- Simple farthest-point sampling with the average keypoint count outperforms many prior few-shot keypoint methods.
- EfficientNet features back-projected outperform CLIP and SAM features for keypoint detection.
Limitations
- Filtering out non-existent keypoints on new shapes and detecting unseen keypoint classes remain difficult.
- Low-quality ShapeNet textures slightly harm performance.
- Multi-view rendering adds computation cost compared to pure 3D methods.
Experimental setup
Datasets: KeypointNet ShapeNet ShapeNet part
Benchmarks: KeypointNet ShapeNet part
Metrics: IoU across geodesic distance thresholds part segmentation IoU
Baselines: SIFT-3D HARRIS-3D ISS D3Feat USIP UKPGAN FSKD NCP
Builds on: DINO PyTorch3D FSKD UKPGAN
Closest papers by meaning
- Few-Shot Geometry-Aware Keypoint Localization (CVPR 2023 · similarity 0.89)
- Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning (NeurIPS 2018 · similarity 0.89)
- Unsupervised 3D Structure Inference from Category-Specific Image Collections (CVPR 2024 · similarity 0.88)
- Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features (NeurIPS 2024 · similarity 0.87)
- Towards Complementary Keypoint Detection via Mixture of Detectors (NeurIPS 2026 · similarity 0.85)
- KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.