Research map / 3D vision / Human pose estimation
Keypoint Fusion for RGB-D Based 3D Hand Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Keypoint-Fusion, an RGB-D method for 3D hand pose estimation that fuses RGB and depth features only at sparse keypoint locations, using each modality to disambiguate the other, reaching SOTA on DexYCB and HO-3D.
Problem
3D hand pose estimation from RGB-D input suffers from the high cost of dense cross-modal feature fusion over many invalid pixels, and from feature ambiguity (color/texture similarity, depth holes and noise) that corrupts cross-modal correlation modeling.
Approach
Restrict cross-modal fusion to keypoint features aggregated around initially predicted joints: heatmap-based and 3D geometry-based weights clean RGB features using depth geometry, RGB features projected into point-cloud space fill depth holes, and a Transformer decoder/encoder then models inter-modal keypoint interactions.
Key results
- On DexYCB S0, Keypoint-Fusion reaches 6.94 mm MPJPE versus 8.03 mm for IPNet (about 13.6% reduction).
- On HO-3D, it achieves 1.79 cm MPJPE, better than IPNet's 1.81 cm, and 1.87 cm after alignment versus 2.01 cm.
- A single fusion stage cuts MPJPE from 11.23 mm (2D CNN baseline) to 7.23 mm, and two stages reach 6.94 mm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the fusion stages from 1 to 3 (ablation) | Keypoint-Fusion · DexYCB S0 | MPJPE (mm): 7.23 -> 6.89 | Helped |
| add RGB KFAM with heatmap-based and geometry weights to the depth-only baseline (ablation) | Keypoint-Fusion · DexYCB S0 | MPJPE (mm): 11.23 -> 7.54 | Helped |
| add projected RGB feature fusion in depth feature aggregation (ablation) | Keypoint-Fusion · DexYCB S0 | MPJPE (mm): 7.17 -> 6.94 | Helped |
| add pose refinement with the depth keypoint feature (ablation) | Keypoint-Fusion · DexYCB S0 | MPJPE (mm): 7.11 -> 6.94 | Helped |
| add the geometry adjacency map to RGB feature aggregation (ablation) | Keypoint-Fusion · DexYCB S0 | MPJPE (mm): 6.97 -> 6.94 | Helped |
| add the heatmap-based weight map to RGB feature aggregation (ablation) | Keypoint-Fusion · DexYCB S0 | MPJPE (mm): 6.96 -> 6.94 | Helped |
| add RGB-D keypoint fusion instead of using the depth-only 2D CNN baseline (ablation) | Keypoint-Fusion · DexYCB S0 | MPJPE (mm): 11.23 -> 6.89 | Helped |
| use Keypoint-Fusion instead of IPNet | 3D hand pose estimation · DexYCB S0 | MPJPE (mm): 8.03 -> 6.94 | Helped |
| use Keypoint-Fusion instead of IPNet | 3D hand pose estimation · HO-3D test | MPJPE (cm): 1.81 -> 1.79 | Helped |
| use Keypoint-Fusion instead of IPNet | 3D hand pose estimation · HO-3D test | MPJPE (cm), aligned: 2.01 -> 1.87 | Helped |
Limitations
- Relies on an initial pose estimate from the depth branch, so later stages depend on its accuracy.
- Depth holes and noise remain a challenge that is only partially addressed by RGB projection.
- Evaluated only on two hand datasets (DexYCB, HO-3D).
Experimental setup
Datasets: DexYCB HO-3D
Benchmarks: DexYCB HO-3D
Metrics: MPJPE PA-MPJPE inference time
Baselines: IPNet SA-Fusion DiffHand HandOccNet METRO A2J AWR Keypoint TR
Builds on: IPNet AWR METRO CMX Transformer
Closest papers by meaning
- Cross-Domain 3D Hand Pose Estimation With Dual Modalities (CVPR 2023 · similarity 0.92)
- Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering (CVPR 2019 · similarity 0.90)
- Dense 3D Regression for Hand Pose Estimation (CVPR 2018 · similarity 0.90)
- Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (CVPR 2018 · similarity 0.90)
- Cross-Modal Deep Variational Hand Pose Estimation (CVPR 2018 · similarity 0.90)
- 3D Hand Shape and Pose Estimation From a Single RGB Image (CVPR 2019 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.