Research map / 3D vision / Human pose estimation
SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a ResNet-50 hand pose encoder with contrastive learning using positive pairs mined as nearest-pose hands from other videos (2.0M Ego4D and 100DOH crops), plus a parameter-free pairwise adaptive weighting in the NT-Xent loss.
Problem
3D hand pose estimation pre-training has used small unlabeled sets and single-image or temporally adjacent positives, limiting the information gained from large in-the-wild hand video collections.
Approach
Mine positive pairs as non-identical hand crops from different videos whose 2D keypoints (PCA-reduced) are most similar, and weight the contrastive loss pairwise by inter-sample keypoint distance within the mini-batch.
Key results
- With Ego&Exo-2M pre-training, SiMHand reaches 15.79 MPJPE on FreiHand vs 18.19 for PeCLR (about 13% lower), 16.71 vs 18.06 on DexYCB, and 18.23 vs 18.88 on AssemblyHands.
- On the Ego-100K setting, adaptive weighting lowers SiMHand MPJPE from 31.06 to 28.84 on FreiHand* (10% labels).
- Mining Top-1 similar hands gives the best result; Top-5000 positives raise MPJPE from 31.06 to 35.34.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with SiMHand on Ego&Exo-2M instead of PeCLR on Ego&Exo-2M | ResNet-50 hand pose estimator · FreiHand | MPJPE: 18.19 -> 15.79 | Helped |
| pre-train with SiMHand on Ego&Exo-2M instead of PeCLR on Ego&Exo-2M | ResNet-50 hand pose estimator · DexYCB | MPJPE: 18.06 -> 16.71 | Helped |
| pre-train with SiMHand on Ego&Exo-2M instead of PeCLR on Ego&Exo-2M | ResNet-50 hand pose estimator · AssemblyHands | MPJPE: 18.88 -> 18.23 | Helped |
| add adaptive weighting (ablation) | SiMHand (Ego-100K) · FreiHand* | MPJPE: 31.06 -> 28.84 | Helped |
| add adaptive weighting (ablation) | SiMHand (Ego-100K) · FreiHand* | PCK-AUC: 68.66 -> 71.07 | Helped |
| add adaptive weighting (ablation) | SimCLR (Ego-100K) · FreiHand* | MPJPE: 53.49 -> 52.58 | Helped |
| add adaptive weighting (ablation) | PeCLR (Ego-100K) · FreiHand* | PCK-AUC: 51.50 -> 53.37 | Helped |
| use a less similar positive hand (Top-2 through Top-100) instead of Top-1 (ablation) | SiMHand (Ego-100K) · FreiHand* | MPJPE: 31.46; 31.85; 31.87; 31.53; 31.54 vs 31.06 | Mixed |
| use a less similar positive hand (Top-500 through Top-5000) instead of Top-1 (ablation) | SiMHand (Ego-100K) · FreiHand* | MPJPE: 32.61; 34.05; 35.34 vs 31.06 | Hurt |
| increase pre-training data from Ego-1M to Ego&Exo-2M (ablation) | SiMHand · FreiHand | MPJPE: 2.28% improvement | Helped |
| increase pre-training data from 50K to 1M (ablation) | SiMHand · FreiHand* | MPJPE: 35.32 to 23.68 | Helped |
| directly train weakly-supervised on noisy 2D keypoints instead of pre-training then fine-tuning | FreiHand* hand pose estimator · FreiHand* | MPJPE: 31.06 -> 61.65 | Hurt |
Surprising findings
- SimCLR pre-training performs worse than training from scratch on FreiHand and DexYCB in the full-data setting.
- Matching pre-training viewpoint (ego vs. exo) to the fine-tuning dataset does not consistently improve performance.
Limitations
- Positive pairs depend on noisy 2D keypoints from an off-the-shelf estimator, which can be unreliable in the wild.
- Gains over pre-training baselines shrink as the amount of labeled fine-tuning data increases.
Experimental setup
Datasets: Ego4D 100DOH FreiHand DexYCB AssemblyHands
Benchmarks: FreiHand DexYCB AssemblyHands
Metrics: MPJPE PCK-AUC
Baselines: SimCLR PeCLR TempCLR w/o pre-training weakly-supervised learning
Builds on: SimCLR PeCLR DetNet NT-Xent
Closest papers by meaning
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.88)
- FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation (CVPR 2025 · similarity 0.88)
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.88)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.87)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.87)
- Hand1000: Generating Realistic Hands from Text with Only 1,000 Images (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.